Data operation verification method and device, equipment and medium

Abstract

The invention relates to a data operation verification method and device, equipment and a medium in the technical field of e-commerce, and the method comprises the steps: obtaining a data operation statement submitted by a user through a single-machine type calculation engine, starting a process, and when the grammar correctness, semantic correctness and quoted database elements of the data operation statement are normal and pass the verification, starting the process; generating a logic execution plan corresponding to the statement; generating a physical execution plan corresponding to the logic execution plan when the verification user has the right to access the database elements required to be referenced and passes the verification; generating a to-be-run code corresponding to the physical execution plan, submitting the to-be-run code to a cluster computing engine for running, obtaining a returned operation result, informing the user, and ending the process; and when any check is not passed, a check abnormity is captured, a user is informed, and the process is ended. Whether data operation passes verification or not can be found in advance in a single-machine operation environment, and invalid data operation is prevented from being processed in a cluster operation environment.

Description

Technical Field

[0001] The present application relates to the field of e-commerce technology, and in particular to a data operation verification method and its corresponding device, computer equipment, and computer-readable storage medium. Background Art

[0002] With the rapid development of big data technology, distributed data processing has become the core means for enterprises to process massive data. In traditional technology, data operation statements provided by users are usually directly submitted to the distributed computing environment for processing. If the data operation statement has parsing exceptions and / or permission verification fails, it takes a relatively long time to return the corresponding exception error information to the user, resulting in an extremely poor user experience.

[0003] In view of the shortcomings of traditional practices, the applicant has been engaged in research in related fields for a long time and has taken a different approach to solve industry problems in the field of e-commerce technology. Summary of the invention

[0004] The primary purpose of the present application is to solve at least one of the above problems and to provide a data operation verification method and its corresponding device, computer equipment, and computer program product.

[0005] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0006] A data operation verification method provided to meet one of the purposes of this application includes the following steps:

[0007] The stand-alone computing engine obtains the data operation statement submitted by the user, starts the process, verifies the grammatical correctness and semantic correctness of the data operation statement, and verifies whether the referenced database elements are normal. If all are normal and pass the verification, a logical execution plan corresponding to the data operation statement is generated;

[0008] Verify whether the user has data access rights corresponding to the database elements referenced in the logical execution plan, and if the user has the data access rights and passes the verification, generate a physical execution plan corresponding to the logical execution plan;

[0009] Generate the code to be executed corresponding to the physical execution plan, submit it to the cluster computing engine for execution, obtain the corresponding operation return result, push it to the user, and end the process;

[0010] When any of the above checks fails, the corresponding check exception detailed text is captured and pushed to the user, ending the process.

[0011] On the other hand, a data operation verification device provided to meet one of the purposes of the present application includes a first plan generation module, a second plan generation module, a first process termination module and a second process termination module, wherein the first plan generation module is used to obtain data operation statements submitted by a user from a stand-alone computing engine, start the process, verify the syntactic correctness, semantic correctness, and whether the referenced database elements of the data operation statements are normal, and when all are normal and pass the verification, generate a logical execution plan corresponding to the data operation statements; the second plan generation module is used to verify whether the user has data access rights corresponding to the database elements referenced in the logical execution plan, and when the user has and passes the verification, generate a physical execution plan corresponding to the logical execution plan; the first process termination module is used to generate a to-be-executed code corresponding to the physical execution plan, submit it to the cluster computing engine for execution, obtain the corresponding operation return result, push it to the user, and terminate the process; the second process termination module is used to capture the corresponding verification exception detailed text when any of the above-mentioned verifications fails, push it to the user, and terminate the process.

[0012] On the other hand, a computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the data operation verification method described in the present application.

[0013] On the other hand, a computer program product provided to meet another purpose of the present application includes a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any embodiment of the present application.

[0014] The technical solution of this application has many advantages, including but not limited to the following aspects:

[0015] The present application firstly, by a stand-alone computing engine, in the parsing stage of the data operation statement provided by the user, identifies whether there are grammatical, semantic errors and abnormalities of database elements, and whether the user has the right to access the required referenced database elements, avoids submitting data operation statements with abnormalities or no access directly to the cluster, reduces the occupation of invalid data operations on the cluster computing engine, thereby greatly improving the effective utilization rate of the cluster computing engine, and at the same time, because it is a pre-check, there is no need to use the cluster computing engine, so that compared with the use of the cluster computing engine for verification, the time consumed is greatly reduced. Secondly, by capturing the problems of verification failure in the process, such as insufficient authority, syntax errors, etc., the user is informed in time to improve the user experience. As a result, the user can find and correct the problem before the data operation statement is submitted to the cluster computing engine for processing, instead of locating the problem through complex log analysis or debugging after the cluster computing engine is running, which greatly improves the efficiency of development and debugging, shortens the time for problem solving, and provides users with a more reliable and efficient data processing environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0017] Figure 1 The network architecture of the e-commerce platform exemplified in this application;

[0018] Figure 2 A flowchart of a typical embodiment of the data operation verification method of the present application;

[0019] Figure 3 This is a principle block diagram of the data operation verification device of the present application;

[0020] Figure 4 A schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0021] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0022] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0023] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0024] like Figure 1 In the network architecture shown, the e-commerce platform 82 is deployed on the Internet to provide corresponding services to its users. Similarly, the devices 80 of the merchant users of the e-commerce platform 82 and the devices 81 of the consumer users are also connected to the Internet to use the services provided by the e-commerce platform.

[0025] The exemplary e-commerce platform 82 provides supply and demand matching of products and / or services to the general public with the help of Internet infrastructure. In the e-commerce platform 82, products and / or services are provided as commodity information. To simplify the description, the concepts of commodity, product, etc. are used in this application to refer to the products and / or services in the e-commerce platform 82, which may specifically be physical products, digital products, tickets, service subscriptions, other offline services, etc.

[0026] In reality, all entities can access the e-commerce platform 82 as users, use various online services provided by the e-commerce platform 82, and achieve the purpose of participating in the business activities achieved by the e-commerce platform 82. These entities can be natural persons, legal persons or social organizations, etc. Corresponding to the two types of entities, merchants and consumers in business activities, the e-commerce platform 82 has two types of users, merchant users and consumer users. All entities in the product distribution chain in business activities, including manufacturers, sellers, retailers, logistics providers, etc., can use online services in the e-commerce platform 82 as merchant users, while consumers in business activities, including real or potential consumers, can use online services in the e-commerce platform 82 as their corresponding consumer users. In actual business activities, the same entity can act as both a merchant user and a consumer user, and this should be understood flexibly.

[0027] The infrastructure used to deploy the e-commerce platform 82 mainly includes the backend architecture and frontend equipment. The backend architecture runs various online services through the service cluster, including middleware or frontend services for the platform, services for consumers, services for merchants, etc., to enrich and improve its service functions; the frontend equipment mainly covers the terminal devices used by users to access the e-commerce platform 82 as clients, including but not limited to various mobile terminals, personal computers, point-of-sale devices, etc. For example, merchant users can enter product information for their online stores through their terminal devices 80, or generate their product information using the interfaces opened by the e-commerce platform; consumer users can access the webpage of the online store implemented by the e-commerce platform 82 through their terminal devices 81, trigger the shopping process through the shopping buttons provided on the webpage, and call various online services provided by the e-commerce platform 82 during the shopping process, so as to achieve the purpose of shopping orders.

[0028] In some embodiments, the e-commerce platform 82 may be implemented by a processing facility including a processor and a memory, the processing facility storing a set of instructions that, when executed, cause the e-commerce platform 82 to perform the e-commerce and support functions involved in the present application. The processing facility may be part of a server, client, network infrastructure, mobile computing platform, cloud computing platform, fixed computing platform, or other computing platform, and provides electronic components, merchant equipment, payment gateways, application developers, marketing channels, transportation providers, customer equipment, point-of-sale equipment, etc. of the e-commerce platform 82.

[0029] The e-commerce platform 82 can be implemented as an online service such as cloud computing service, software as a service (SaaS), infrastructure as a service (IaaS), platform as a service (PaaS), desktop as a service (DaaS), hosted software as a service, mobile backend as a service (MBaaS), information technology management as a service (ITMaaS), etc. In some embodiments, the various functional components of the e-commerce platform 82 can be implemented to be suitable for operation on various platforms and operating systems. For example, for an online store, its administrator users enjoy the same or similar functions regardless of various embodiments such as iOS, Android, HomonyOS, or web pages.

[0030] The e-commerce platform 82 can realize the corresponding independent station for each merchant to run its corresponding online store, and provide the merchant with the corresponding business management engine instance for the merchant to establish, maintain and run one or more online stores in one or more independent stations. The business management engine instance can be used for content management, task automation and data management of one or more online stores, and various specific business processes of the online store can be configured through interfaces or built-in components to support the implementation of business activities. The independent station is the infrastructure of the e-commerce platform 82 with cross-border service functions. Merchants can maintain their online stores more centrally and autonomously based on the independent station. The independent station usually has a domain name and storage space dedicated to the merchant, and different independent stations are relatively independent. The e-commerce platform 82 can provide standardized or personalized technical support for a large number of independent stations, so that merchant users can customize their own business management engine instance and use this business management engine instance to maintain one or more online stores they own.

[0031] The online store can implement backend configuration and maintenance by having the merchant user log in to its business management engine instance as an administrator. With the support of various online services provided by the infrastructure of the e-commerce platform 82, the merchant user can configure various functions in its online store as an administrator, view various data, etc. For example, the merchant user can manage various aspects of its online store, such as viewing the online store's recent activities, updating the online store's product catalog, managing orders, recent visit activities, total order activities, etc.; the merchant user can also view more detailed information about the business and visitors to the merchant's online store by obtaining reports or metrics, such as showing a sales summary of the merchant's overall business, specific sales and participation data of active sales marketing channels, etc.

[0032] The e-commerce platform 82 may provide communication facilities and associated merchant interfaces for providing electronic communications and marketing, such as utilizing electronic message aggregation facilities to collect and analyze communication interactions between merchants, consumers, merchant devices, customer devices, point-of-sale devices, etc., aggregating and analyzing communications, such as for increasing the potential for providing product sales, etc. For example, a consumer may have a question related to a product, which may generate a dialogue between the consumer and the merchant (or an automated processor-based agent on behalf of the merchant), where the communication facility is responsible for interacting and providing the merchant with analysis on how to increase the probability of a sale.

[0033] In some embodiments, an application suitable for installation in a terminal device can be provided to serve the access needs of different users, so that various users can access the e-commerce platform 82 by running the application in the terminal device, such as the merchant backend module of the online store in the e-commerce platform 82, etc. In the process of implementing business activities through these functions, the e-commerce platform 82 can implement various functions related to supporting the implementation of business activities as middleware or online services and open corresponding interfaces, and then implant the toolkit corresponding to the interface access function into the application to implement function expansion and task implementation. The business management engine can include a series of basic functions, and expose these functions to online services and / or application calls through APIs. Online services and applications use corresponding functions by remotely calling corresponding APIs.

[0034] With the support of various components of the business management engine instance, the e-commerce platform 82 can provide online shopping functions, enabling merchants to establish connections with customers in a flexible and transparent manner. Consumer users can select items online, create product orders, provide the delivery address of the goods in the product order, and complete the payment confirmation of the product order. Then, the merchant can review and complete or cancel the order. The review component carried by the business management engine instance can realize the compliance use of business processes to ensure that the order is suitable for fulfillment before actual fulfillment. Orders may sometimes be fraudulent and need to be verified (such as ID card checks). There is a payment method that requires merchants to wait to ensure that funds are received, which can play a role in preventing such risks, etc. Order risks may be generated by fraud detection tools submitted by third parties through order risk APIs, etc. Before fulfillment, merchants may need to obtain payment information or wait to receive payment information in order to mark the order as paid, so that the merchant can prepare to deliver the product. Such situations can be reviewed accordingly. The review process can be implemented by the fulfillment component. Merchants can use the fulfillment component to review, adjust work, and trigger related fulfillment services, such as: manual fulfillment services, which are used when merchants pick and pack products in boxes, purchase shipping labels and enter their tracking numbers, or simply mark items as fulfilled; custom fulfillment services, which can define email notifications; API fulfillment services, which can trigger third-party applications to create fulfillment records in third parties; legacy fulfillment services, which can trigger custom API calls from the business management engine to third parties; gift card fulfillment services. You can provide the generation of numbers and activate gift cards. Merchants can use the order printer application to print shipping orders. The fulfillment process can be executed when the items are packed in boxes and ready for shipping, tracking, delivery, consumer receipt verification, etc.

[0035] It can be seen that the services provided by the e-commerce platform are based on products. The corresponding commodity data is the basic data of the e-commerce platform. The commodity information is provided through the commodity data. The mining and utilization of commodity data is the basis for realizing various technical services, including the use of user transaction data and commodity data in the commodity data of the e-commerce platform to provide basic services for the operation of the data processing system. Therefore, the data processing system can run on any one or more servers of the e-commerce platform's cluster, so as to realize various functions by using the various commodity data provided by the e-commerce platform.

[0036] A data operation verification method of the present application can be programmed as a computer program product and deployed in a client or server for execution. For example, in the exemplary application scenario of the present application, it can be deployed and implemented in the server of an e-commerce customer service platform, thereby accessing an interface opened after the computer program product is running and performing human-computer interaction with the process of the computer program product through a graphical user interface to execute the method.

[0037] See also Figure 2 The data operation verification method of the present application, in its typical embodiment, comprises the following steps:

[0038] Step S1100: The stand-alone computing engine obtains the data operation statement submitted by the user, starts the process, verifies the grammatical correctness and semantic correctness of the data operation statement, and verifies whether the referenced database elements are normal. If all are normal and pass the verification, a logical execution plan corresponding to the data operation statement is generated;

[0039] The stand-alone computing engine is a mode in which Spark is deployed and run on a single computer, also known as Spark on Local mode. In this mode, Spark deployment requires that the environment configuration of Hadoop and Yarn is not performed when Spark is installed on the computer.

[0040] The Spark is a fast and general computing engine designed for large-scale data processing. It improves the speed of data processing through in-memory computing, which makes it very advantageous in processing big data tasks that require fast response.

[0041] Users can submit their written data operation statements to the stand-alone computing engine through methods such as Spark-shell or Spark-submit. The data operation statements are represented by SQL statements, which are a programming language specifically used to operate relational databases and can perform various operations on data in the database, such as query, insert, update and delete.

[0042] The stand-alone computing engine receives the data operation statement submitted by the user and starts the process. During this process, the data operation statement is converted multiple times and finally compiled into a code to be executed in preparation for its execution. Specifically, first, the Parser component in the Spark SQL of the stand-alone computing engine parses the data operation statement. During this process, the bottom layer uses the Antlr framework, for example, Antlr4 generates a lexical parser: SqlBaseLexer and a syntax parser: SqlBaseParser according to the syntax rules of the SQL statement defined in SqlBase.g4. SqlBaseLexer performs lexical analysis on the data operation statement, parses each keyword and various identifiers in the data operation statement, separates each word from the data operation statement: Token, and obtains a Token stream. SqlBaseParser performs syntax analysis on the data operation statement. When the syntax of the data operation statement is correct, the Token stream is used to construct a syntax tree: AST, and AstBuilder converts the constructed AST into an unresolved logical plan: Unresolved Logical Plan is an abstract execution plan that describes operations such as data input, output, conversion, and filtering, but does not contain specific execution details and is not bound to fields and data types. When there is an error in the syntax of the data operation statement, an exception description text of the syntax parsing of the data operation statement is thrown accordingly.

[0043] The Analyzer component in the Spark SQL analyzes the Unresolved Logical Plan generated above. In this process, the pre-defined Catalyst Rule and SessionCatelog information are used to perform relevant transform operations on the UnresolvedLogical Plan, and further semantic analysis is performed to verify the data structure, schema, data type, etc., that is, to verify whether the table name and column name in the database element required to be referenced by the data operation statement exist. Then, when the semantics are correct and the table name and column name in the referenced database element exist and the reference is normal, the analyzed logical plan obtained by successfully performing the transform operation on the Unresolved Logical Plan is obtained: Analyzed Logical Plan, which is also called the resolved logical plan: Plan, as the logical execution plan, for example: UnresolvedRelation[ITEMS], UnresolvedRelation[ORDERS] The relations marked as Unresolved will be processed into actual table names and actual column names in this process; when the semantics of the data operation statement is incorrect or the referenced database element is abnormal, the description text of the analysis exception of the data operation statement is thrown accordingly.

[0044] The SessionCatalog is mainly used for the unified management of various function resource information and metadata information (database, data table, data view, data partition and function, etc.). If the metadata comes from Hive, the metadata stored in MySQL will be pulled into the SessionCatalog.

[0045] Step S1200, verifying whether the user has data access rights corresponding to the database elements referenced in the logical execution plan, and if the user has the rights and passes the verification, generating a physical execution plan corresponding to the logical execution plan;

[0046] It can be understood that Spark SQL permission control can be implemented based on Ranger. Usually, resource authentication rules, row-level filtering rules, and data desensitization rules are implemented based on the Ranger data stack. The specific configuration of each rule can be implemented in the Web UI front-end interface or back-end code programming provided by Ranger. In addition, these rules are registered and effective in the Optimizer stage of SQL parsing through the Spark SQL Extensions mechanism. The resource authentication rules are mainly used to control the access rights of users or user groups to resources such as tables and columns in the database. Administrators can set which users or user groups have access to specific tables or columns in the database through the Web UI front-end interface provided by Ranger. The row-level filtering rules are mainly used to control the access rights of users or user groups to rows in the database. Similarly, administrators can set which users or user groups have access to specific rows in the database through the Web UI front-end interface provided by Ranger.

[0047] The Ranger is an open source permission management framework that can provide secure access control to the Hadoop ecosystem. Secondly, Ranger provides developers with an extensible framework that can perform unified data security management, with built-in access control for multiple components including Hadoop, Hive, HBase, Kafka, etc.

[0048] After the logical execution plan corresponding to the data operation statement is generated, the Optimizer stage begins, calling and verifying whether the user has the data access rights corresponding to the tables, columns, and rows in the database elements referenced in the logical execution plan according to the resource authentication rules and / or row-level filtering rules, as well as the user's identity. When the user has the data access rights corresponding to the tables, columns, and rows, the Optimizer component in the Spark SQL applies a series of optimization rules to the logical execution plan, rewrites the inefficient structure in the logical execution plan while ensuring that the corresponding final result is correct, and obtains the optimized logical plan corresponding to the logical execution plan after optimization: Optimized Logical Plan. Most of the series of optimization rules are heuristic rules, that is, rules derived based on intuition or experience, such as: column pruning, predicate pushdown, constant accumulation, constant replacement, and trying to evaluate the filter to optimize the query before executing the Project (projection query), etc. The user's identity is unique, usually the user's ID, and can be set as needed by technicians in this field.

[0049] The Planner component in the Spark SQL applies various physical plan strategies to the nodes corresponding to the OptimizedLogical Plan to generate multiple candidate physical plans. The cost corresponding to each candidate physical plan is analyzed based on the Cost model evaluation, and the physical plan with the minimum cost is selected as the Physical Plan, which is used as the physical execution plan.

[0050] When the user does not have the data access permission corresponding to the table, column or row, an exception description text of the access permission of the data operation statement is thrown.

[0051] Step S1300: Generate the code to be executed corresponding to the physical execution plan, submit it to the cluster computing engine for execution, obtain the corresponding operation return result, push it to the user, and end the process;

[0052] The cluster computing engine deploys and runs Spark on a cluster consisting of multiple computers. This mode is also called Spark on YARN mode. In this mode, Spark, as an application of YARN, can use the resource management and scheduling capabilities provided by YARN to optimize resource allocation and task execution in the cluster.

[0053] The Planner component continues to prepare for the physical execution plan, such as ensuring the correctness of partition operations, node reuse in the physical execution plan, running code generation, etc., and finally generates RDD as the code to be run.

[0054] The code to be run is submitted to the cluster computing engine for execution. The Spark scheduler distributes the code to be run to different nodes in the cluster for execution. Finally, the corresponding execution results are summarized to form the final operation return result, which is pushed to the user. In this way, the data operation required by the user is completed and the result is returned to the user, thus ending the process.

[0055] Step S1400: When any of the above verifications fails, the corresponding verification exception detailed text is captured and pushed to the user, and the process ends.

[0056] It can be understood that the exception description text of the grammatical analysis of the data operation statement thrown out by each of the above-mentioned verification failures, the description text of the analysis exception of the data operation statement, and the exception description text of the access rights of the data operation statement can all be captured to obtain these description texts. Therefore, when any of the above-mentioned verifications fails, the corresponding captured description text is used as the verification exception description text and pushed to the user to provide the user with clear feedback and clarify the corresponding exception details. In addition, since the verification fails, the process is interrupted and cannot continue to be executed, and the process ends.

[0057] It can be known from the typical embodiments of the present application that the technical solution of the present application has many advantages, including but not limited to the following aspects:

[0058] The present application firstly, by a stand-alone computing engine, in the parsing stage of the data operation statement provided by the user, identifies whether there are grammatical, semantic errors and abnormalities of database elements, and whether the user has the right to access the required referenced database elements, avoids submitting data operation statements with abnormalities or no access directly to the cluster, reduces the occupation of invalid data operations on the cluster computing engine, thereby greatly improving the effective utilization rate of the cluster computing engine, and at the same time, because it is a pre-check, there is no need to use the cluster computing engine, so that compared with the use of the cluster computing engine for verification, the time consumed is greatly reduced. Secondly, by capturing the problems of verification failure in the process, such as insufficient authority, syntax errors, etc., the user is informed in time to improve the user experience. As a result, the user can find and correct the problem before the data operation statement is submitted to the cluster computing engine for processing, instead of locating the problem through complex log analysis or debugging after the cluster computing engine is running, which greatly improves the efficiency of development and debugging, shortens the time for problem solving, and provides users with a more reliable and efficient data processing environment.

[0059] In a further embodiment, after step S1300, the code to be executed corresponding to the physical execution plan is generated and before it is submitted to the cluster computing engine for execution, the following steps are included:

[0060] Step S2300: judging whether any one or more of nested operations, table join operations, aggregation operations, grouping operations, and multi-predicate filtering operations are required according to the logic execution plan;

[0061] Determine whether the logical execution plan contains nested operations. The nested operations usually involve subqueries, that is, one query contains another query, and identify whether such a nested relationship exists. When the logical execution plan contains a dependency on the result of another query, it means that a nested operation is required;

[0062] Determine whether the logical execution plan involves the connection of multiple tables. For example, the data in different tables are merged together according to specific conditions through the Join operation. When the logical execution plan contains references to multiple tables, and these tables need to be associated through the Join operation, then this indicates that a table join operation is required.

[0063] Determine whether the logical execution plan contains aggregate functions, such as sum: SUM, average: AVG, maximum value: MAX, minimum value: MIN, etc., and whether to group the data: GROUP BY. When the logical execution plan contains any multiple of these aggregate functions and / or grouping operations, it means that the corresponding aggregate operation and / or grouping operation is required;

[0064] It is determined whether the logical execution plan contains multiple conditions in the WHERE clause. These conditions may involve different columns and operators. When the logical plan contains multiple filtering conditions, it means that a multi-predicate filtering operation is required.

[0065] Step S2310: When the above operations are not needed, the stand-alone computing engine runs the code to be run, obtains the corresponding operation return result, pushes it to the user, and ends the process.

[0066] When the above operations are not needed, it means that the stand-alone computing engine is likely to be able to run the code to be run, and generally there will be no memory overflow: OOM (Out of Memory). Therefore, the code to be run is handed over to the stand-alone computing engine for execution, so as to obtain the corresponding operation return result, which is pushed to the user. Thus, the data operation required by the user is completed, and the result is returned to the user, thus ending the process.

[0067] When any of the above operations is required, the cluster computing engine runs the code to be run, obtains the corresponding operation return result, pushes it to the user, and ends the process.

[0068] In this embodiment, by adding a judgment step for the logic execution plan before the stand-alone computing engine runs the code to be executed, this step can identify whether the data operation statement requires complex nested operations, table join operations, aggregation operations, grouping operations or multi-predicate filtering operations, so that it can more intelligently decide whether it is necessary to mobilize the resources of the cluster computing engine. When the logic execution plan does not require these high-running resource operations, the stand-alone computing engine is generally capable of processing these data operation statements, thereby greatly reducing the running time. At the same time, due to the reduced dependence on cluster running resources, the running cost can be reduced, providing users with faster and more reliable data processing services.

[0069] In a further embodiment, after step S1300, the code to be executed corresponding to the physical execution plan is generated and before it is submitted to the cluster computing engine for execution, the following steps are included:

[0070] Step S3300: determining the total amount of data that needs to be referenced by the data operation statement according to the physical execution plan;

[0071] It can be understood that each operation node in the physical execution plan contains information about the amount of data it involves processing. For example, when the operation involves reading the data of the entire table, the size of the table is the amount of data that needs to be considered. When the operation involves filtering or aggregation, the amount of data required may vary depending on the filtering conditions or aggregation keys. Therefore, based on this information in the physical execution plan, the amount of data required for each operation node can be accumulated to obtain the total amount of data required for the entire data operation statement.

[0072] Step S3310: When the total amount of data does not exceed the preset threshold, the stand-alone computing engine runs the code to be run, obtains the corresponding operation return result, pushes it to the user, and ends the process.

[0073] When the total amount of data does not exceed the preset threshold, it means that the stand-alone computing engine is likely to be able to run the code to be run, and generally there will be no memory overflow: OOM (Out of Memory). Therefore, the code to be run is handed over to the stand-alone computing engine to run, so as to obtain the corresponding operation return result, which is pushed to the user, thereby completing the data operation required by the user and returning the result to the user, thus ending the process. The preset threshold can be flexibly set by those skilled in the art as needed.

[0074] When the total amount of data exceeds a preset threshold, the cluster computing engine runs the code to be run, obtains the corresponding operation return result, pushes it to the user, and ends the process.

[0075] In this embodiment, by determining whether the total amount of data that needs to be referenced by the data operation statement is high, if it is not very high, it can be quickly handed over to the stand-alone computing engine to process the returned results and push them to the user, greatly reducing the running time. At the same time, due to the reduced dependence on cluster operating resources, the operating cost can be reduced, providing users with faster and more reliable data processing services.

[0076] In a further embodiment, step S3310, running the code to be run by a stand-alone computing engine, includes the following steps:

[0077] Step S3311, obtaining the running time of the stand-alone computing engine running the code to be run;

[0078] It can be understood that a timer initialized to zero can be constructed. When the stand-alone computing engine receives the code to be run, the timing is started and the corresponding running time is accumulated until the stand-alone computing engine returns the corresponding running result or the stand-alone computing engine is interrupted from running the code to be run, and the timing ends, and the accumulated running time is reset to zero.

[0079] Step S3312: when the running time exceeds the preset time, the stand-alone computing engine is interrupted from running the code to be run, and the code is transferred to the cluster computing engine for running, and the corresponding operation return result is obtained and pushed to the user, and the process ends;

[0080] When the running time exceeds the preset time, it means that the stand-alone computing engine is not efficient enough to run the code to be run or there is a fault in the stand-alone computing engine. Therefore, the code to be run is transferred to the cluster computing engine for running, and the corresponding operation return result is obtained and pushed to the user, ending the process. The preset time can be flexibly set by those skilled in the art as needed.

[0081] In this embodiment, when the stand-alone computing engine runs inefficiently or fails, it can be switched to the cluster computing engine in a timely manner, thereby ensuring the robustness of data processing.

[0082] In a further embodiment, step S1200, verifying whether the user has data access rights corresponding to the database elements referenced in the logic execution plan, comprises the following steps:

[0083] Step S1210, obtaining the user group to which the user belongs;

[0084] It is understandable that administrators can configure the user group to which each user belongs in the Web UI front-end interface provided by Ranger, which can manage user permissions more conveniently and efficiently. The setting of user groups helps to simplify the permission management process, because administrators only need to set permissions for user groups instead of configuring them for each user separately.

[0085] Step S1220, calling the preconfigured access authentication rule to verify whether the user group has the right to access any one or multiple of the tables, columns, and rows included in the database elements referenced in the logic execution plan, and correspondingly determining whether the verification is passed.

[0086] The access authentication rules may be resource authentication rules and / or row-level rules. By configuring the resource authentication rules, the permission control of user groups accessing tables and columns in the database can be implemented. By configuring the row-level filtering rules, the permission control of user groups accessing rows in the database can be implemented.

[0087] It is not difficult to understand that after the logical execution plan corresponding to the data operation statement is generated, the Optimizer stage begins, calling the pre-configured access authentication rules, and checking whether the user group has the right to access the corresponding items in the tables, columns, and rows of the database elements referenced in the logical execution plan according to the access authentication rules and the group identifier of the user group. Thus, when the user group has the right to access each corresponding item, it is confirmed that the users belonging to the user group have passed the verification; when the user group has no right to access any of the corresponding items, it is confirmed that the users belonging to the user group have not passed the verification. The group identifier of the user group is unique, and can usually be the ID of the user group, which can be set as needed by those skilled in the art.

[0088] In this embodiment, the process of verifying the user's data access rights is disclosed to ensure that only users with corresponding permissions can access or operate specific data resources, thereby enhancing data security. Secondly, it can facilitate the centralized management of permissions for multiple users of the same type.

[0089] In a further embodiment, after step S1300, the code to be executed corresponding to the physical execution plan is generated and before it is submitted to the cluster computing engine for execution, the following steps are included:

[0090] Step S4300, obtaining a sentence decision model that has been pre-trained to a convergence state;

[0091] The statement decision model is pre-trained to a convergence state, and learns the ability of a target computing engine to run the code to be run after generating the code to be run corresponding to the data operation statement from a stand-alone computing engine and a cluster computing engine according to the data operation statement decision.

[0092] Step S4310, calling the statement decision model, and according to the data operation statement provided by the user, deciding from the stand-alone computing engine and the cluster computing engine the target computing engine that runs the corresponding code to be run, and having the target computing engine run the code to be run, obtaining the corresponding operation return result, pushing it to the user, and ending the process.

[0093] The data operation statement is input into the statement decision model, the corresponding deep semantic information is extracted, and the deep semantic information is vectorized to represent the corresponding decision basis vector. The deep semantic information indicates that the computing resources required for processing the data operation statement are suitable to be provided by a stand-alone computing engine or a cluster computing engine. In this process, the model will use the pre-learned statement analysis knowledge, which may include any of the length, complexity, amount of data involved, and operation type of the data operation statement, to evaluate the computing resources of the input data operation statement for decision-making. Further, the decision basis vector is subjected to binary classification mapping to determine the classification probability of the category of the data operation statement processed by the stand-alone computing engine and the classification probability of the category of the data operation statement processed by the cluster computing engine. The largest of the two classification probabilities is selected, and its corresponding computing engine is used as the target computing engine. The code to be run corresponding to the data operation statement is run to obtain the corresponding operation return result, which is pushed to the user to end the process.

[0094] In this embodiment, by adopting a statement decision model, the computing engine that best processes data operation statements can be predicted from stand-alone computing engines and cluster computing engines, so that computing resources can be allocated quickly, accurately and reasonably, processing performance can be optimized, and the user's data operation statements can be executed in the most efficient manner.

[0095] In a further embodiment, before step S4300, obtaining the sentence decision model pre-trained to a converged state, the following steps are included:

[0096] Step S5300: collect data operation statements corresponding to the codes to be run by the stand-alone computing engine and add them to the first data set;

[0097] Construct a first data set initialized to be empty, and construct a first counter initialized to zero, which is used to accumulate the total number of difference statements in the first data set. Then, each time a single code to be run is handed over to a stand-alone computing engine for execution, except for the first time, obtain the data operation statement corresponding to the code to be run and add it to the first data set, and determine the difference between the data operation statement and each data operation statement in the first data set. When any difference exceeds the first preset threshold, the total number of difference statements accumulated by the first counter increases by one; when all differences do not exceed the first preset threshold, the number of difference statements accumulated by the first counter does not increase. In addition, at the first time, directly add the data operation statement corresponding to the code to be run to the first data set, and directly increase the total number of difference statements accumulated by the first counter by one. The first preset threshold can be set by a person skilled in the art as needed.

[0098] Step S5310: collect data operation statements corresponding to a plurality of codes to be run by the cluster computing engine and add them to a second data set;

[0099] Construct a second data set initialized to be empty, and construct a second counter initialized to zero, which is used to accumulate the total number of difference statements in the second data set. Then, each time a single code to be run is handed over to the cluster computing engine for execution, except for the second time, obtain the data operation statement corresponding to the code to be run and add it to the second data set, and determine the difference between the data operation statement and each data operation statement in the second data set. When any difference exceeds the second preset threshold, the total number of difference statements accumulated by the second counter increases by one; when all differences do not exceed the second preset threshold, the number of difference statements accumulated by the second counter does not increase. In addition, in the second time, the corresponding code to be run is directly added to the second data set, and the total number of difference statements accumulated by the second counter is directly increased by one. The second preset threshold can be set by a person skilled in the art as needed.

[0100] The degree of difference between the two data operation statements can be determined by using a large language model, or by pre-modeling the difference between the two data operation statements based on a deep learning algorithm, and learning to quantify the degree of the difference. The specific implementation can be flexibly adapted by those skilled in the art.

[0101] Step S5320: when the total number of difference sentences in the first data set and the total number of difference sentences in the second data set both meet the requirements, construct a training set based on the first data set and the second data set;

[0102] When the total number of difference statements in the first data set and the total number of difference statements in the second data set both exceed the third preset threshold, it means that the total number of these two difference statements have met the standard. Then, each data operation statement in the first data set is taken as a single training sample, and the supervisory label of the training sample is annotated to represent that the data operation statement is processed by a stand-alone computing engine without being processed by a cluster computing engine, for example: [1,0]; and each data operation statement in the second data set is taken as a single training sample, and the supervisory label of the training sample is annotated to represent that the data operation statement is processed by a cluster computing engine without being processed by a stand-alone computing engine, for example: [0,1].

[0103] Step S5330: Use the training set to train the statement decision model until it converges, so that it can learn the ability of the target computing engine to run the code to be run after generating the code to be run corresponding to the data operation statement from the stand-alone computing engine and the cluster computing engine according to the data operation statement decision.

[0104] The sentence decision model includes a text representation layer and a classification layer. The recommended selection for the text representation layer is the BERT model. Other models such as Fasttext, TextCNN, DPCNN, TextRCNN, TextBiLSTM, etc. may also be used. The classification layer may be a single-layer fully connected layer or an MLP (multi-layer perceptron).

[0105] A single training sample and its supervision label in the training set are called, and the training sample is input into the sentence decision model for forward reasoning, and the corresponding deep semantic information is extracted, and the deep semantic information is vectorized to represent the corresponding decision basis vector, and the decision basis vector is mapped into two categories, and the classification probability of the category of the data operation statement processed by the stand-alone computing engine and the classification probability of the category of the data operation statement processed by the cluster computing engine are predicted, and the two predicted classification probabilities constitute the prediction result, and the preset cross entropy loss function or mean square error loss function is called, which can be flexibly changed by technicians in this field according to prior knowledge or experimental experience. Through setting, the loss value of the prediction result is calculated according to the supervisory label of the training sample. When the loss value is less than or equal to the convergence threshold, it indicates that the sentence decision model has been trained to a convergence state, so that the training of the sentence decision model can be terminated; when the loss value exceeds the convergence threshold, it indicates that the sentence decision model has not converged, so the sentence decision model is gradient updated according to the loss value, usually by back propagation to correct the weight parameters of each link of the sentence decision model so that the sentence decision model is closer to convergence, and then continue to call other training samples and their supervisory labels to perform iterative training on the sentence decision model until the sentence decision model is trained to a convergence state. The convergence threshold can be set by a person skilled in the art as needed.

[0106] In this embodiment, the training process of the sentence decision model is disclosed to ensure the accuracy and reliability of the reasoning results of the model.

[0107] See also Figure 3, a data operation verification device provided to meet one of the purposes of the present application is a functional embodiment of the data operation verification method of the present application. On the other hand, the device is a data operation verification device provided to meet one of the purposes of the present application, including a first plan generation module 1100, a second plan generation module 1200, a first process end module 1300 and a second process end module 1400, wherein the first plan generation module 1100 is used to obtain the data operation statement submitted by the user by a stand-alone computing engine, start the process, verify the grammatical correctness, semantic correctness of the data operation statement, and whether the referenced database elements are normal. When they are all normal and pass the verification, the data is generated. A logical execution plan corresponding to the operation statement; a second plan generation module 1200, used to verify whether the user has the data access rights corresponding to the database elements referenced in the logical execution plan, and when the user has the rights and passes the verification, a physical execution plan corresponding to the logical execution plan is generated; a first process ending module 1300, used to generate the code to be executed corresponding to the physical execution plan, submit it to the cluster computing engine for execution, obtain the corresponding operation return result, push it to the user, and end the process; a second process ending module 1400, used to capture the corresponding verification exception detailed text when any of the above verifications fails, push it to the user, and end the process.

[0108] In a further embodiment, the first process ending module 1300 includes: an operation judgment submodule, which is used to judge whether any one or more of nested operations, joint table operations, aggregation operations, grouping operations, and multi-predicate filtering operations are required according to the logical execution plan; a first process ending submodule, which is used to, when the above operations are not required, run the code to be run by a stand-alone computing engine, obtain the corresponding operation return result, push it to the user, and end the process.

[0109] In a further embodiment, the first process ending module 1300 includes: a total amount determination submodule, used to determine the total amount of data that needs to be referenced by the data operation statement according to the physical execution plan; a second process ending submodule, used to, when the total amount of data does not exceed a preset threshold, run the code to be run by a stand-alone computing engine, obtain the corresponding operation return result, push it to the user, and end the process.

[0110] In a further embodiment, the stand-alone computing engine runs the code to be run, including: a time-consuming acquisition submodule, used to obtain the running time of the stand-alone computing engine running the code to be run; a third process ending submodule, used to interrupt the running of the code to be run by the stand-alone computing engine when the running time exceeds a preset time length, transfer it to the cluster computing engine for running, obtain the corresponding operation return result, push it to the user, and end the process;

[0111] In a further embodiment, the second plan generation module 1200 includes: a user group acquisition submodule, used to obtain the user group to which the user belongs; a result acquisition submodule, used to call preconfigured access authentication rules to verify whether the user group has the right to access any one or more of the tables, columns, and rows included in the database elements referenced in the logical execution plan, and correspondingly determine whether the verification has passed.

[0112] In a further embodiment, the first process ending module 1300 includes: a model acquisition submodule, used to obtain a statement decision model pre-trained to a convergence state; a fourth process ending submodule, used to call the statement decision model, and according to the data operation statement provided by the user, decide from the stand-alone computing engine and the cluster computing engine to run the corresponding target computing engine to be run. The target computing engine runs the code to be run, obtains the corresponding operation return result, pushes it to the user, and ends the process.

[0113] In a further embodiment, before the model acquisition submodule, it includes: a first statement adding submodule, which is used to collect multiple data operation statements corresponding to the code to be run run by the stand-alone computing engine and add them to the first data set; a second statement adding submodule, which is used to collect multiple data operation statements corresponding to the code to be run run by the cluster computing engine and add them to the second data set; a training set construction submodule, which is used to construct a training set based on the first data set and the second data set when the total number of difference statements in the first data set and the total number of difference statements in the second data set meet the standards; a model training submodule, which is used to use the training set to train the statement decision model until it converges, so that it can learn the ability of the target computing engine to run the code to be run after generating the code to be run corresponding to the data operation statement from the stand-alone computing engine and the cluster computing engine.

[0114] In order to solve the above technical problems, the present application also provides a computer device. Figure 4As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a data operation verification method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the data operation verification method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] In this embodiment, the processor is used to execute Figure 3 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the data operation verification device of this application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0116] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the data operation verification method of any embodiment of the present application.

[0117] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0118] In summary, the present application can detect in advance whether a data operation has passed verification in a single-machine operating environment, thereby avoiding invalid data operations in a cluster operating environment.

[0119] It will be understood by those skilled in the art that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, changed, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, changed, rearranged, decomposed, combined, or deleted. Furthermore, the steps, measures, and schemes in the various operations, methods, and processes in the prior art that are open source and in this application may also be alternated, changed, rearranged, decomposed, combined, or deleted.

[0120] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data operation verification method, characterized in that: The steps include: The stand-alone computing engine obtains the data operation statement submitted by the user, starts the process, verifies the grammatical correctness and semantic correctness of the data operation statement, and verifies whether the referenced database elements are normal. If all are normal and pass the verification, a logical execution plan corresponding to the data operation statement is generated; Verify whether the user has data access rights corresponding to the database elements referenced in the logical execution plan, and if the user has the data access rights and passes the verification, generate a physical execution plan corresponding to the logical execution plan; Generate the code to be executed corresponding to the physical execution plan, submit it to the cluster computing engine for execution, obtain the corresponding operation return result, push it to the user, and end the process; When any of the above checks fails, the corresponding check exception detailed text is captured and pushed to the user, ending the process.

2. The data operation verification method according to claim 1, characterized in that: After the code to be executed corresponding to the physical execution plan is generated and before it is submitted to the cluster computing engine for execution, the following steps are included: Determine whether any one or more of nested operations, table join operations, aggregation operations, grouping operations, and multi-predicate filtering operations are required according to the logic execution plan; When the above operations are not needed, the stand-alone computing engine runs the code to be run, obtains the corresponding operation return result, pushes it to the user, and ends the process.

3. The data operation verification method according to claim 1, characterized in that: After the code to be executed corresponding to the physical execution plan is generated and before it is submitted to the cluster computing engine for execution, the following steps are included: Determine the total amount of data that the data operation statement needs to reference according to the physical execution plan; When the total amount of data does not exceed the preset threshold, the stand-alone computing engine runs the code to be run, obtains the corresponding operation return result, pushes it to the user, and ends the process.

4. The data operation verification method according to claim 1, characterized in that: Running the code to be run by a stand-alone computing engine includes the following steps: Obtaining the running time of the stand-alone computing engine running the code to be run; When the running time exceeds the preset time, the stand-alone computing engine is interrupted to run the code to be run, and the code is transferred to the cluster computing engine for running, and the corresponding operation return result is obtained and pushed to the user, and the process ends.

5. The data operation verification method according to claim 1, characterized in that: Verifying whether the user has data access rights corresponding to the database elements referenced in the logical execution plan includes the following steps: Get the user group to which the user belongs; The preconfigured access authentication rules are called to verify whether the user group has the right to access any one or multiple of the tables, columns, and rows included in the database elements referenced in the logic execution plan, and correspondingly determine whether the verification is passed.

6. The data operation verification method according to claim 1, characterized in that: After the code to be executed corresponding to the physical execution plan is generated and before it is submitted to the cluster computing engine for execution, the following steps are included: Obtain a sentence decision model that has been pre-trained to a convergent state; The statement decision model is called, and according to the data operation statement provided by the user, a target computing engine that runs the corresponding code to be run is determined from the stand-alone computing engine and the cluster computing engine. The target computing engine runs the code to be run, obtains the corresponding operation return result, pushes it to the user, and ends the process.

7. The data operation verification method according to claim 6, characterized in that: Before obtaining the sentence decision model that has been pre-trained to a converged state, the following steps are included: Collecting a plurality of data operation statements corresponding to the codes to be executed by the stand-alone computing engine and adding them to the first data set; Collecting a plurality of data operation statements corresponding to the codes to be executed by the cluster computing engine and adding them to a second data set; When the total number of difference sentences in the first data set and the total number of difference sentences in the second data set both meet the requirements, constructing a training set based on the first data set and the second data set; The statement decision model is trained with the training set until it converges, so that it can learn the ability of the target computing engine to run the code to be run after generating the code to be run corresponding to the data operation statement according to the data operation statement decision from the stand-alone computing engine and the cluster computing engine.

8. A data operation verification device, characterized in that: include: The first plan generation module is used to obtain the data operation statement submitted by the user by the stand-alone computing engine, start the process, verify the grammatical correctness and semantic correctness of the data operation statement, and whether the referenced database elements are normal. When all are normal and pass the verification, generate a logical execution plan corresponding to the data operation statement; A second plan generation module is used to verify whether the user has data access rights corresponding to the database elements referenced in the logical execution plan, and if the user has the data access rights and passes the verification, generate a physical execution plan corresponding to the logical execution plan; The first process ending module is used to generate the to-be-executed code corresponding to the physical execution plan, submit it to the cluster computing engine for execution, obtain the corresponding operation return result, push it to the user, and end the process; The second process ending module is used to capture the corresponding verification exception detailed text when any of the above verifications fails, push it to the user, and end the process.

9. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images