A data processing method, system, apparatus, device and storage medium

Abstract

The application relates to the technical field of data processing and artificial intelligence, and provides a data processing method, system, device, equipment and storage medium to generate an execution plan with good effect for a data processing request in an HTAP service scenario, and improve processing efficiency. In the application, a received data processing request is analyzed to determine a corresponding request target; a target processing engine is selected based on the request target; the target processing engine is a transaction processing engine or an analysis processing engine; an execution plan is generated for the data processing request through a target optimizer in the target processing engine; the data processing request is processed according to the execution plan through a target executor in the target processing engine; when the target processing engine is a transaction processing engine, the target optimizer is a transaction processing optimizer, and the target executor is a transaction processing executor; when the target processing engine is an analysis processing engine, the target optimizer is an analysis processing optimizer, and the target executor is an analysis processing executor.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more particularly to the field of artificial intelligence technology, providing a data processing method, system, apparatus, device, and storage medium. Background Technology

[0002] Today, data-intensive applications often use large amounts of data for various real-time tasks (e.g., data-driven decision-making), thus requiring the combination of Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP).

[0003] In response, much recent academic and industrial effort has focused on developing Hybrid Transactional and Analytical Processing (HTAP) systems that provide optimal OLTP and OLAP services. Under this technology, the data management system of an HTAP system consists of OLTP executors, OLAP executors, and a unified optimizer. The unified optimizer generates execution plans for both the OLTP and OLAP executors.

[0004] In practical applications, due to the significant differences in the applicable rules between OLTP and OLAP executors, the unified optimizer cannot achieve universality between the two. Specifically, when the unified optimizer performs well for one executor, its performance for the other is uncontrollable. In this case, the unified optimizer cannot select the best execution plan for all data processing requests, resulting in the generation of poor execution plans for some data processing requests. This, in turn, affects the efficiency of the executor in processing data according to the generated execution plan.

[0005] In summary, in HTAP service scenarios, how to generate effective execution plans for each received data processing request and improve data processing efficiency is an urgent problem to be solved. Summary of the Invention

[0006] This application provides a data processing method, system, apparatus, device, and storage medium for generating effective execution plans for each received data processing request in an HTAP service scenario, thereby improving data processing efficiency.

[0007] This application provides a data processing method, the method comprising:

[0008] The received data processing requests are parsed and processed to determine the corresponding request target;

[0009] Based on the request objective, select the target processing engine; the target processing engine can be a transaction processing engine or an analytical processing engine.

[0010] The target optimizer in the target processing engine generates an execution plan for the data processing request.

[0011] The data processing requests are processed according to the execution plan through the target executor in the target processing engine;

[0012] When the target processing engine is a transaction processing engine, the target optimizer is a transaction processing optimizer, and the target executor is a transaction processing executor; when the target processing engine is an analysis processing engine, the target optimizer is an analysis processing optimizer, and the target executor is an analysis processing executor.

[0013] This application provides a data processing system, which includes: a parser, a classifier, a transaction processing engine, and an analysis processing engine; the transaction processing engine includes: a transaction processing optimizer and a transaction processing executor; the analysis processing engine includes: an analysis processing optimizer and an analysis processing executor; wherein:

[0014] The parser is used to: parse and process received data processing requests to determine the corresponding request target;

[0015] The classifier is used to select a target processing engine based on the request target; wherein the target processing engine is either a transaction processing engine or an analysis processing engine.

[0016] The target optimizer in the target processing engine is used to: generate execution plans for data processing requests;

[0017] The target executor in the target processing engine is used to process data processing requests according to the execution plan;

[0018] When the target processing engine is a transaction processing engine, the target optimizer is a transaction processing optimizer, and the target executor is a transaction processing executor; when the target processing engine is an analytic processing engine, the target optimizer is an analytic processing optimizer, and the target executor is an analytic processing executor.

[0019] This application provides a data processing apparatus, the apparatus comprising:

[0020] The determining unit is used to parse and process the received data processing request and determine the corresponding request target;

[0021] The selection unit is used to select a target processing engine based on the request target; the target processing engine can be a transaction processing engine or an analysis processing engine.

[0022] The generation unit is used to generate an execution plan for the data processing request through the target optimizer in the target processing engine;

[0023] The processing unit is used to process data processing requests according to the execution plan through the target executor in the target processing engine;

[0024] When the target processing engine is a transaction processing engine, the target optimizer is a transaction processing optimizer, and the target executor is a transaction processing executor; when the target processing engine is an analysis processing engine, the target optimizer is an analysis processing optimizer, and the target executor is an analysis processing executor.

[0025] In one possible implementation, the request target includes: a data operation type; the selection unit is specifically used for:

[0026] If the data operation type indicates that the data processing request is a non-read-only query, then the transaction processing engine is selected as the target processing engine.

[0027] In one possible implementation, the request target includes: data operation type and data processing information; the selection unit is specifically used for:

[0028] If the data operation type indicates that the data processing request is a read-only query, then the target processing engine is selected for the data processing request based on the data processing information and the database environment information.

[0029] In one possible implementation, the data processing information includes data table information and condition information; the selection unit is specifically used for:

[0030] If the data table information indicates that data on a single data table needs to be processed, and based on the data access column in the condition information and the column definition of the data access column in the database environment information, it is determined that the data processing request is a single table lookup processing request, then the transaction processing engine is selected as the target processing engine.

[0031] If the data table information indicates that data on multiple data tables needs to be processed, and the condition information indicates that there is a connection relationship between the multiple data tables, then the target processing engine is selected for the data processing request based on the data volume and data distribution information of the data tables in the database environment information.

[0032] In one possible implementation, the selected unit is specifically used for:

[0033] If there is a target data table with a data volume less than the data volume threshold among multiple data tables, and the data distribution information indicates that the data on the target data table meets the distribution conditions, then the transaction processing engine is selected as the target processing engine.

[0034] If there is a target data table with a data volume less than the data volume threshold among multiple data tables, and the data distribution information indicates that the data on the target data table does not meet the distribution conditions, then the analysis and processing engine is selected as the target processing engine.

[0035] In one possible implementation, after selecting a target processing engine for the data processing request based on the request objective, and before generating an execution plan for the data processing request through the target optimizer in the target processing engine, the selection unit is further used for:

[0036] Based on the data operation status, the performance of the processing engine selected based on the request target is verified, and the corresponding verification results are obtained. The verification results indicate whether the target execution performance of the processing engine selected based on the request target reaches the preset performance index.

[0037] Based on the verification results, when the target execution performance reaches the preset performance index, the target processing engine is determined to be used.

[0038] In one possible implementation, the data operation state includes at least one of the following: data storage state, data synchronization state, and data resource state;

[0039] Among them, data storage status refers to the stored data read from the underlying storage medium in the cache resource pool;

[0040] Data synchronization status indicates the degree of synchronization between the data modifications made by the transaction processing engine and the data replicas used by the analytical processing engine through an asynchronous synchronization mechanism.

[0041] Data resource status indicates the remaining amount of computing and memory resources.

[0042] In one possible implementation, if the processing engine selected based on the request target is an analysis processing engine, the data running state includes the data storage state; the selection unit is further used for:

[0043] The analysis and processing engine identifies the scanned data from the data access columns that were scanned. This scanned data is then matched against the stored data to obtain the matching results. The matching results represent the hit rate of the scanned data in the stored data.

[0044] If the hit rate is not less than the preset hit threshold, a verification result is obtained indicating that the target execution performance has reached the preset performance index.

[0045] In one possible implementation, if the processing engine selected based on the request target is an analysis processing engine, the data running state includes the data synchronization state; the selection unit is further used for:

[0046] Based on the data synchronization status, if it is determined that the data modifications processed by the transaction processing engine are completely synchronized to the data copy used by the analysis processing engine, then a verification result indicating that the target execution performance has reached the preset performance index is obtained.

[0047] In one possible implementation, if the processing engine selected based on the request target is an analysis processing engine, the data running state includes the data resource state; the selection unit is further used for:

[0048] The estimated analysis and processing engine calculates the target amount of resources required to process the data processing request. If the target amount of resources is not greater than the remaining resources, a verification result is obtained indicating that the target execution performance has reached the preset performance index.

[0049] In one possible implementation, the selection unit is also used for:

[0050] Based on the verification results, if the target execution performance does not meet the preset performance indicators, use other processing engines to replace the processing engine selected based on the request target, and use the replaced processing engine as the target processing engine when processing data processing requests.

[0051] In one possible implementation, after selecting a target processing engine for the data processing request based on the request objective, and before generating an execution plan for the data processing request through the target optimizer in the target processing engine, the selection unit is further used for:

[0052] Based on historical anomaly data, the performance of the processing engine selected based on the request target is verified, and the corresponding verification results are obtained. The verification results indicate whether the target execution performance of the processing engine selected based on the request target meets the preset performance indicators.

[0053] Based on the verification results, when the target execution performance reaches the preset performance index, the target processing engine is determined to be used.

[0054] Among them, historical anomaly data includes at least the correspondence between historical processing requests and the actual response time of the historical processing engine.

[0055] In one possible implementation, the selected unit is used for:

[0056] The target classifier is used to select a target processing engine for the data processing request based on the request target; the target classifier is obtained by training on the data processing request with processor engine annotations.

[0057] An electronic device provided in this application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the above-described data processing methods.

[0058] In one possible implementation, when a computer program is executed by a processor, the processor performs the following process:

[0059] The received data processing requests are parsed and processed to determine the corresponding request target;

[0060] Based on the request objective, a target processing engine is selected for the data processing request; the target processing engine is either a transaction processing engine or an analytical processing engine.

[0061] The target optimizer in the target processing engine generates an execution plan for the data processing request.

[0062] The data processing requests are processed according to the execution plan through the target executor in the target processing engine;

[0063] When the target processing engine is a transaction processing engine, the target optimizer is a transaction processing optimizer, and the target executor is a transaction processing executor; when the target processing engine is an analysis processing engine, the target optimizer is an analysis processing optimizer, and the target executor is an analysis processing executor.

[0064] This application provides a computer-readable storage medium including a computer program. When the computer program is run on an electronic device, the computer program is used to cause the electronic device to perform the steps of any of the above-described data processing methods.

[0065] This application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. When the processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the above-described data processing methods.

[0066] The beneficial effects of this application are as follows:

[0067] This application provides a data processing method, system, apparatus, device, and storage medium, relating to the field of data processing technology, and particularly to the fields of database technology and artificial intelligence technology.

[0068] In this application, the data processing system in the HTAP scenario includes a transaction processing engine for OLTP (Online Transaction Processing) with a transaction processing optimizer and a transaction processing executor, and an analysis processing engine for OLAP (Online Analytical Processing) with an analysis processing optimizer and an analysis processing executor. In this case, each optimizer serves its matched executor, and each executor processes data based on the execution plan generated by its matched optimizer. This avoids the situation where a unified optimizer cannot achieve universality between two executors, resulting in the inability to generate a satisfactory execution plan. In other words, for each received data processing request, the executor can generate a satisfactory execution plan for its associated optimizer, thereby improving data processing efficiency.

[0069] In the data processing implementation provided in this application:

[0070] First, the received data processing request is parsed to determine the corresponding request target. By parsing the data processing request, the request target can be accurately obtained, and then processing can be carried out on the request target, ensuring that the data processing request can be accurately understood and executed efficiently.

[0071] Then, based on the request target, a target processing engine is selected from the transaction processing engine and the analytical processing engine. The target optimizer within the target processing engine then generates an execution plan for the data processing request. Finally, the target executor within the target processing engine processes the data processing request according to the execution plan. By intelligently selecting the target processing engine based on the request target, utilizing its optimizer to generate an execution plan, and then using its executor to process according to the plan, data processing requests can be processed accurately and efficiently. This approach not only improves execution efficiency but also optimizes resource utilization and reduces latency, thereby enhancing overall system performance and user experience. This ensures processing quality while accelerating response speed and enhancing system flexibility and scalability.

[0072] In summary, in the HTAP service scenario, the data processing system generates an effective execution plan for each received data processing request, ensuring that the received data processing requests are executed accurately and without error, thereby improving data processing efficiency and overall performance.

[0073] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0074] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0075] Figure 1 This is a schematic diagram of an application scenario in an embodiment of this application;

[0076] Figure 2 This is a schematic diagram of a data processing scenario provided in an embodiment of this application;

[0077] Figure 3 This application provides a schematic diagram of a data processing system architecture.

[0078] Figure 4 A flowchart of the first data processing method provided in the embodiments of this application;

[0079] Figure 5 A schematic diagram of a target selection processing engine based on a request target provided in an embodiment of this application;

[0080] Figure 6 A flowchart of a second data processing method provided in an embodiment of this application;

[0081] Figure 7 This is a schematic diagram illustrating a first method for performance verification of a processing engine selected based on a request target, as provided in an embodiment of this application.

[0082] Figure 8 This is a schematic diagram of a data status synchronization process provided in an embodiment of this application;

[0083] Figure 9 This is a schematic diagram illustrating a second method for performance verification of a processing engine selected based on a request target, as provided in an embodiment of this application.

[0084] Figure 10 A schematic diagram illustrating the first method of determining the target processing engine used in data processing, as provided in an embodiment of this application;

[0085] Figure 11 This is a schematic diagram illustrating the storage of historical anomaly data provided in an embodiment of this application;

[0086] Figure 12 A flowchart of a third data processing method provided in the embodiments of this application;

[0087] Figure 13 A schematic diagram illustrating a second method for determining the target processing engine used in data processing, as provided in an embodiment of this application;

[0088] Figure 14A structural diagram of a data processing apparatus provided in an embodiment of this application;

[0089] Figure 15 This is a structural diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0090] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0091] The following describes some of the concepts involved in the embodiments of this application.

[0092] 1. Online Transaction Processing (OLTP):

[0093] OLTP is a data processing technology focused on supporting high-frequency, small-scale transaction processing, emphasizing real-time response, high concurrency, and data consistency and integrity. It is widely used in various business scenarios requiring immediate processing and high reliability, such as bank account transactions, online shopping order processing on e-commerce platforms, and flight bookings. Therefore, OLTP is specifically designed to support and manage transaction-oriented applications.

[0094] 2. Online Analytical Processing (OLAP):

[0095] OLAP is a data processing technology that allows users to quickly perform complex, multi-dimensional analytical queries. OLAP systems are designed to support decision support systems, business intelligence applications, and other business scenarios that require complex analysis of large amounts of data, such as enterprise data analysis and mining, sales analysis, financial analysis, market research, and supply chain management. It's important to note that, compared to OLTP, OLAP focuses on data analysis rather than transaction processing, enabling rapid aggregation and statistical operations on multiple columns of data.

[0096] 3. Hybrid Transactional and Analytical Processing (HTAP):

[0097] HTAP is a database management system architecture designed to integrate OLTP (Online Transaction Processing) that supports high-throughput transaction processing with OLAP (Online Transaction Processing) that supports low-latency analytics. HTAP breaks down the boundaries between using dedicated OLTP for day-to-day business operations and separate OLAP for complex data analysis. HTAP allows for the efficient execution of both types of tasks within the same database management system, enabling a single database management system to efficiently handle real-time transactions and real-time analytics. It features unified storage, parallel processing, hybrid columnar and row-based storage, intelligent resource scheduling, and real-time data analytics capabilities. It is particularly suitable for enterprise applications that require both high-performance transaction processing and real-time data analytics, such as financial services, e-commerce, logistics and supply chain management, and IoT data analytics.

[0098] 4. Target Processing Engine:

[0099] The target processing engine is the core component of the HTAP database management system, containing key components such as optimizers and executors. In this embodiment, since HTAP services integrate OLTP and OLAP services, the HTAP database management system includes a transaction processing engine for OLTP services and an analysis processing engine for OLAP services. The target processing engine is one of these two engines. The transaction processing engine includes a transaction optimizer and a transaction executor, while the analysis processing engine includes an analysis optimizer and an analysis executor.

[0100] 5. Optimizer:

[0101] The optimizer, also known as the query optimizer, is a core component of a database management system. It is responsible for determining the most efficient way to execute data processing requests, analyzing various possible execution plans, and selecting the one with the lowest cost. Specifically, it generates multiple potential execution plans based on the specific details of the data processing request, and evaluates the expected execution cost of different plans based on database environment information such as the data volume, metadata, and statistics of the data tables. From all feasible execution plans, it selects the one with the lowest expected execution cost as the final execution plan. In practical applications, the optimizer directly impacts query performance; a good optimization strategy can significantly reduce query response time and resource consumption. In this embodiment, since HTAP services integrate OLTP and OLAP services, the HTAP database management system includes a transaction processing optimizer for OLTP services and an analytical processing optimizer for OLAP services.

[0102] 6. Actuator:

[0103] The executor, also known as the query executor, is primarily responsible for executing queries according to the execution plan selected by the optimizer. The executor implements physical-level operations such as scanning tables, using indexes to find specific records, sorting, performing joins, and aggregations. The executor also handles transaction management to ensure atomicity, consistency, isolation, and durability, and manages concurrency control and locking mechanisms. Furthermore, the executor generates result sets, extracting data according to the execution plan and returning it to the client. In practical applications, the executor directly affects the actual execution efficiency of queries and the stability and reliability of the system. In this embodiment, since HTAP services integrate OLTP and OLAP services, the HTAP database management system includes a transaction processing executor for the OLTP service and an analytical processing executor for the OLAP service.

[0104] 7. Implementation Plan:

[0105] In a database management system (DBMS) context, an execution plan is a detailed strategy generated by the DBMS's query optimizer for a data processing request. The execution plan is a series of steps or operations. These steps describe how the DBMS will actually execute the data processing request, including which tables to access, which indexes to use, whether to sort, the join method, and the return results to the client. The execution plan encompasses the entire set of logical and physical operations from the input data processing request to the actual execution and retrieval of the required data, guiding the DBMS on how to efficiently execute the data processing request.

[0106] 8. Data Processing System:

[0107] A data processing system, also known as a database management system (DBMS), is a software system used to create, manage, and maintain databases, and to provide access and manipulation functions for data. It is one of the core components of modern information systems and is widely used in various fields such as enterprise, finance, and healthcare. In this embodiment, since HTAP services integrate OLTP and OLAP services, the HTAP database management system includes a database supporting OLTP services and a data warehouse supporting OLAP services.

[0108] A database is an organized collection of data, typically stored electronically in a computer system. It allows users to easily access, manage, and update data. Primarily used in OLTP (Online Transaction Processing) systems, it supports routine operations such as banking transactions and hotel bookings. It requires fast response times to handle a large number of short queries, such as inserting, updating, or deleting single records. In this application embodiment, the database includes, but is not limited to, relational database management systems such as MySQL, PostgreSQL, and Oracle.

[0109] A data warehouse is a data storage system specifically designed for analysis and reporting. It collects data from multiple heterogeneous data sources and transforms it into a consistent format for in-depth analysis. Primarily used for OLAP, it supports complex querying and data analysis tasks, helping decision-makers gain insights. It is suitable for batch loading large amounts of historical data and executing complex queries such as aggregation and grouping. In the embodiments of this application, data warehouses include, but are not limited to: Amazon Redshift, Google BigQuery, Snowflake, Teradata, IBM Db2 Warehouse, etc.

[0110] 9. Target Classifier

[0111] The target classifier is used to select the target processing engine for data processing requests, so as to enable more refined and faster data management and access control.

[0112] The word “exemplary” as used below means “serving as an example, embodiment, or illustration.” Any embodiment illustrated as an “exemplary” need not be construed as superior to or better than other embodiments.

[0113] The terms "first" and "second" used in this document are for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more features; for example, "first connection relationship" and "second connection relationship" in the embodiments of this application are used in a road segment connection relationship table determined based on two different types of information.

[0114] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more. For example, if there are multiple sample questions in the embodiments of this application, it means that there are two or more sample questions in the input question-answering model of this application.

[0115] The design concept of the embodiments of this application is briefly introduced below:

[0116] Different data management systems have their own applicable service scenarios, mainly divided into OLTP and OLAP service scenarios. As user needs continue to become more diverse, data-intensive applications often use large amounts of data for various real-time tasks (e.g., data-driven decision-making). At this point, a single OLTP service scenario or a single OLAP service scenario can no longer meet user needs, so it is necessary to combine OLTP and OLAP.

[0117] In response, many academic and industrial efforts have recently focused on developing HTAP systems that provide optimal OLTP and OLAP services. Under related technologies, the data management system of an HTAP system consists of OLTP executors, OLAP executors, and a unified optimizer. The unified optimizer generates execution plans for both the OLTP and OLAP executors.

[0118] In practical applications, due to the significant differences in the applicable rules between OLTP and OLAP executors, the unified optimizer cannot achieve universality between the two. Specifically, when the unified optimizer performs well for one executor, its performance for the other is uncontrollable. In this case, the unified optimizer cannot select the best execution plan for all data processing requests, resulting in the generation of poor execution plans for some data processing requests. This, in turn, affects the efficiency of the executor in processing data according to the generated execution plan.

[0119] In summary, in HTAP service scenarios, how to generate effective execution plans for each received data processing request and improve data processing efficiency is an urgent problem to be solved.

[0120] This application provides a data processing method, system, apparatus, device, and storage medium, relating to the field of data processing technology, and particularly to the fields of database technology and artificial intelligence technology.

[0121] In this application, the data processing system in the HTAP scenario includes a transaction processing engine for OLTP (Online Transaction Processing) with a transaction processing optimizer and a transaction processing executor, and an analysis processing engine for OLAP (Online Analytical Processing) with an analysis processing optimizer and an analysis processing executor. In this case, each optimizer serves its matched executor, and each executor processes data based on the execution plan generated by its matched optimizer. This avoids the situation where a unified optimizer cannot achieve universality between two executors, resulting in the inability to generate a satisfactory execution plan. In other words, for each received data processing request, the executor can generate a satisfactory execution plan for its associated optimizer, thereby improving data processing efficiency.

[0122] In the data processing implementation provided in this application:

[0123] First, the received data processing request is parsed to determine the corresponding request target. During the parsing process, the data processing request is checked for syntax and analyzed for semantics. Syntax checking ensures the syntactic accuracy of the data processing request, and semantic analysis identifies the request target and its logical validity. Therefore, parsing the data processing request accurately obtains the request target, and then the request target is processed accordingly.

[0124] Then, based on the request target, a target processing engine is selected from the transaction processing engine and the analytical processing engine. The target optimizer within the target processing engine then generates an execution plan for the data processing request. Finally, the target executor within the target processing engine processes the data processing request according to the execution plan. By intelligently selecting the target processing engine based on the request target, utilizing its optimizer to generate an execution plan, and then using its executor to process according to the plan, data processing requests can be processed accurately and efficiently. This approach not only improves execution efficiency but also optimizes resource utilization and reduces latency, thereby enhancing overall system performance and user experience. This allows for faster response times while ensuring data processing quality, and enhances the system's flexibility and scalability.

[0125] In summary, in the HTAP service scenario, the data processing system generates an effective execution plan for each received data processing request, ensuring that the received data processing requests are executed accurately and without error, thereby improving data processing efficiency and overall performance.

[0126] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0127] See Figure 1 , Figure 1 This is a schematic diagram of an application scenario in an embodiment of this application. The application scenario includes a terminal device 110 and a server 120.

[0128] In one alternative implementation, the terminal device 110 and the server 120 can communicate via a communication network. The communication network can be a wired network or a wireless network.

[0129] Therefore, terminal device 110 and server 120 can be connected directly or indirectly via wired or wireless communication. For example, terminal device 110 can be indirectly connected to server 120 via a wireless access point, or terminal device 110 can be directly connected to server 120 via the Internet; this application does not impose any restrictions on this.

[0130] In this application embodiment, the terminal device 110 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, vehicle terminals, and other devices; the terminal device may have a client related to data processing functions installed, which may be software, such as a browser with data processing functions, an artificial intelligence (AI) tool with data processing functions, or a webpage, a mini-program, etc.

[0131] In this embodiment, the server is equipped with database management system functions. Server 120 is a backend server corresponding to the client, or a server specifically used for data processing; this application does not impose specific limitations. Server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0132] It should be noted that the data processing methods in the various embodiments of this application can be executed by an electronic device, which can be a terminal device 110 or a server 120. That is, the method can be executed by the terminal device 110 or the server 120 alone, or by the terminal device 110 and the server 120 together.

[0133] For example, when executed jointly by terminal device 110 and server 120, a user can trigger a data processing request through a client installed on terminal device 110. After receiving the data processing request, server 120 first parses and processes the received data processing request to determine the corresponding request target; then, based on the request target, it selects a target processing engine for the data processing request; the target processing engine can be a transaction processing engine or an analytical processing engine; finally, through the target optimizer in the target processing engine, it generates an execution plan for the data processing request; and through the target executor in the target processing engine, it processes the data processing request according to the execution plan.

[0134] It should be noted that, Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and is not specifically limited in the embodiments of this application.

[0135] In this embodiment of the application, when there are multiple servers, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; the data involved in the data processing method disclosed in this embodiment of the application can be stored on the blockchain, such as database environment information, data operation status, historical abnormal data, etc.

[0136] Furthermore, the embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, database technology, artificial intelligence, cloud computing, distributed computing, healthcare, e-commerce, financial services, logistics and supply chain management, etc. Specifically, they can be applied to data processing-related application scenarios in these fields; a few examples are briefly listed below:

[0137] (I) E-commerce scenario.

[0138] In e-commerce scenarios, websites need to process orders in real time and also analyze customer behavior to optimize recommendation systems. (See also...) Figure 2 , Figure 2 This is a schematic diagram illustrating a data processing scenario provided in an embodiment of this application. For example... Figure 2 As shown, orders are processed in real time by interacting with the server through terminal devices, and the processing results are fed back to the terminal devices.

[0139] In this scenario, a user can trigger an order processing SQL statement using a client on their terminal device. The terminal device then sends the order processing request to the corresponding server. The server parses and processes the SQL statement using its data processing system to obtain the request target. Based on the target, it selects the target processing engine from the transaction processing engine and the analytical processing engine. Assuming the transaction processing engine is selected as the target processing engine, the order processing request is sent to it for processing. Within the transaction processing engine, the transaction optimizer generates an execution plan for the SQL statement, and the transaction executor processes the order according to this plan. Finally, the processing result is fed back to the terminal device, allowing the user to view the result through the client on the terminal device.

[0140] (II) Logistics and Supply Chain Management Scenarios.

[0141] In logistics and supply chain management scenarios, the HTAP database management system monitors the status of goods in real time and makes adjustments to strategies based on the latest information.

[0142] It should be noted that the application scenarios listed above are just simple examples, and other application scenarios are also applicable to the embodiments of this application, which will not be elaborated here.

[0143] It should be emphasized that the specific embodiments of this application involve user-related data, such as data processing requests input during data processing and the data requested for processing. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the target user is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0144] To avoid the problem of reduced data processing efficiency in HTAP service scenarios where the unified optimizer cannot achieve universality between OLTP and OLAP executors, resulting in the inability to generate effective execution plans for various received data processing requests, this application proposes a data processing system.

[0145] See Figure 3 , Figure 3 This application provides an embodiment of a data processing system architecture diagram; the data processing system includes: a parser, a classifier, a transaction processing engine, and an analysis processing engine. The transaction processing engine further includes a transaction processing optimizer and a transaction processing executor, and the analysis processing engine further includes an analysis processing optimizer and an analysis processing executor; wherein:

[0146] The parser is used to parse and process received data processing requests to determine the corresponding request target.

[0147] For example, the parser performs lexical and syntactic analysis on the received data processing request to obtain the Abstract Syntax Tree (AST) corresponding to the data processing request; wherein, the AST is used to describe the logical structure of the data processing request, and each node in the AST represents the request target in the data processing request.

[0148] During lexical analysis, the data processing request is read word by word, each token is identified, and each token is converted into an internal representation of the data processing system. The tokens include, but are not limited to, keywords, identifiers, operators, and delimiters. The keywords are the request targets of the data processing request, which include, but are not limited to, data table information, conditional information (such as query conditions and join conditions), selection columns, and data operation types.

[0149] During syntax analysis, the data processing request is verified to conform to the predefined syntax rules, and the symbols obtained from lexical analysis are matched according to the predefined syntax rules to gradually build a tree structure, namely the abstract syntax tree.

[0150] In one possible implementation, after the parser performs syntactic analysis to determine syntactic correctness, it also performs semantic analysis. Specifically, it verifies the validity and legitimacy of each request target in the data processing request by accessing the system directory of the data processing system; for example, it confirms whether the table name and column names exist, whether the data type matches, and whether the permissions are sufficient. This step helps ensure that the data processing request is not only formally correct but also logically feasible.

[0151] In this embodiment, the data processing request can be a Structured Query Language (SQL) statement. SQL statements are a standard language used to manage and manipulate data processing systems. Through SQL statements, data processing such as data querying, inserting, updating, modifying, deleting, and joining and aggregating multiple data tables can be performed on data or data tables in the data management system for HTAP service scenarios.

[0152] When parsing data processing requests, the parser breaks them down into multiple symbols through lexical analysis and then constructs an abstract syntax tree according to grammatical rules through syntax analysis. This ensures the syntactic accuracy of the data processing requests. Furthermore, semantic analysis identifies the request targets and their logical validity within the data processing requests. Therefore, parsing data processing requests accurately reveals the request targets, enabling processing based on those targets and ensuring that the data processing requests are accurately understood and executed efficiently.

[0153] The classifier is used to select a target processing engine based on the request target; the target processing engine is either a transaction processing engine or an analysis processing engine.

[0154] It should be noted that the implementation method of selecting the target processing engine based on the request target is described in the specific implementation section of the data processing method, and will not be elaborated here.

[0155] The target optimizer in the target processing engine is used to: generate execution plans for data processing requests; when the target processing engine is a transaction processing engine, the target optimizer is a transaction processing optimizer; when the target processing engine is an analytical processing engine, the target optimizer is an analytical processing optimizer.

[0156] In one possible implementation, the target optimizer generates an execution plan by executing steps A1-A4:

[0157] In step A1, the target optimizer transforms or optimizes the received data processing requests based on preset rules and heuristic algorithms, generating new processing requests that are logically equivalent but have higher execution efficiency. This reduces computational complexity, improves data access efficiency, or simplifies query structures. For example, target optimization might convert data processing requests into JOIN operations or apply techniques such as view merging and constant folding to simplify data processing.

[0158] In step A2, the target optimizer generates multiple potential execution plans based on the data processing request and its rewritten new processing request. Each potential execution plan represents the processing logic to be executed for the data processing request; multiple potential execution plans represent completing the same data processing request through various different processing logics.

[0159] In step A3, the target optimizer estimates the cost of multiple potential execution plans generated based on database environment information such as the data volume, metadata, and statistics of the data table, and obtains the corresponding expected execution cost.

[0160] The amount of data in a data table includes, but is not limited to: the total number of rows in the data table, the number of columns involved in data processing, the data type and size of each column;

[0161] Meta information includes, but is not limited to: table definitions, column definitions, and index information of the data tables involved in the data processing request;

[0162] Statistical information includes, but is not limited to: data distribution (such as the degree of data skewness), number of unique values ​​(NDV), cardinality information, contour histogram, singleton histogram, and rough set; among which, data distribution can be used as a reference for the number of unique values ​​in a column.

[0163] The expected execution costs include, but are not limited to: input / output (I / O) costs, central processing unit (CPU) costs, and network transmission costs. Among these, I / O costs involve the number of disk accesses; CPU costs involve computational complexity, such as the CPU time required for operations like comparison and sorting; and network transmission costs involve the overhead of data transfer across nodes in a distributed system.

[0164] Step A4: The objective optimizer selects the execution plan with the lowest expected execution cost from all potential execution plans.

[0165] In this embodiment of the application, the transaction processing optimizer can be a traditional optimizer or an AI optimizer, and the analysis processing optimizer can be a traditional optimizer or an AI optimizer.

[0166] Traditional optimizers employ two main optimization methods: rule-based optimization (RBO) and cost-based optimization (CBO).

[0167] RBO (Responsive Business Optimizer) generates an execution plan based on a series of predefined rules. CBO (Consistent Cost Optimizer) estimates the cost of the entire query execution plan based on the cost of a series of predefined basic operations, and selects the lowest-cost execution plan from the possible plans. CBO mainly consists of three parts: cardinality estimation, cost model, and plan enumeration. It uses database environment information to calculate the cost of each operation step and selects the lowest-cost plan from the enumerated plans.

[0168] The AI ​​optimizer primarily uses artificial intelligence technology, based on the execution plan training dataset, to optimize the cardinality estimation, cost calculation, and plan enumeration of traditional optimizers. The AI ​​optimizer uses machine learning models to continuously learn and analyze new data, adapting to changes in data distribution and query patterns. In this embodiment, the AI ​​optimizer for OLTP scenarios is trained based on the execution plan of the OLTP executor, and the AI ​​optimizer for OLAP scenarios is trained based on the execution plan of the OLAP executor.

[0169] Throughout the process of generating the execution plan, the target optimizer relies on detailed database environment information and complex algorithms to ensure that the selected execution plan can provide the fastest response speed with minimal resource consumption, so that even when faced with very complex queries, the data processing system can complete the task as efficiently as possible.

[0170] The target executor in the target processing engine is used to: process data processing requests according to the execution plan; when the target processing engine is a transaction processing engine, the target executor is a transaction processing executor; when the target processing engine is an analysis processing engine, the target executor is an analysis processing executor.

[0171] In one possible implementation, the target executor processes the data by performing steps B1-B4:

[0172] Step B1: The target executor prepares the runtime environment and database environment information for the task to be executed; wherein, the runtime environment includes, but is not limited to: memory allocation, opening necessary files or connections;

[0173] Step B2: The target executor parses the execution plan and determines the specific execution steps and sequence;

[0174] Step B3: The target executor performs physical operations on the data processing request according to the steps and order obtained from the parsing; physical operations include, but are not limited to: scan operations, filter operations, projection operations, join operations, aggregation operations, sorting operations, and grouping operations;

[0175] In step B4, the target executor obtains the data processing results, formats the results according to the client's requirements, and then sends them back to the client.

[0176] The target actuator performs various physical operations step by step until the data processing result is generated. This process ensures that the data processing can be completed efficiently and accurately, while maintaining the consistency and integrity of the data. At the same time, the data processing system can effectively support complex data operation requirements.

[0177] In this application, the data processing system in the HTAP scenario includes a transaction processing engine for OLTP (Online Transaction Processing) with a transaction processing optimizer and a transaction processing executor, and an analysis processing engine for OLAP (Online Analytical Processing) with an analysis processing optimizer and an analysis processing executor. In this case, each optimizer serves its matched executor, and each executor processes data based on the execution plan generated by its matched optimizer. This avoids the situation where a unified optimizer cannot achieve universality between two executors, resulting in the inability to generate a satisfactory execution plan. In other words, for each received data processing request, the executor can generate a satisfactory execution plan for its associated optimizer, thereby improving data processing efficiency.

[0178] based on Figure 3 The data processing system shown in this application provides a data processing method, see below. Figure 4 , Figure 4 The first data processing method provided in the embodiments of this application includes the following steps:

[0179] Step S400: Parse the received data processing request to determine the corresponding request target.

[0180] It should be noted that the specific implementation of step S400 can be found in the description of the parser in the above data processing system, and will not be repeated here.

[0181] Step S401: Select a target processing engine based on the request target; the target processing engine is either a transaction processing engine or an analysis processing engine.

[0182] In this embodiment of the application, when selecting a target processing engine based on the request target, the data operation type is first read from the request target, and then the target processing engine is selected from the transaction processing engine and the analysis processing engine according to the data operation type.

[0183] When selecting the target processing engine based on the data operation type, if the data operation type indicates that the data processing request is a non-read-only query, then the transaction processing engine is selected as the target processing engine; if the data operation type indicates that the data processing request is a read-only query, then the data processing information is read from the request target, and based on the data processing information and the database environment information, the target processing engine is selected from the transaction processing engine and the analytical processing engine for the data processing request.

[0184] The data table processing information includes, but is not limited to: data table information (such as data table identifier, number of data tables), and condition information (such as query conditions on data tables, join conditions between data tables);

[0185] Database environment information includes, but is not limited to: the amount of data in the data table (such as the number of rows of records in the data table involved in the data processing request), metadata (such as the table definition, column definition, index information, etc. of the data table involved in the data processing request), and statistical information (such as data distribution, number of unique values, cardinality information, contour histogram, singleton histogram, rough set, etc.).

[0186] Based on data table processing information and database environment information, when selecting a target processing engine from transaction processing engines and analytical processing engines for a data processing request, there are several implementation methods as follows:

[0187] Example 1: Data table information indicates that data on a single data table needs to be processed. Based on the data volume of each data table in the database environment information, if the data volume of the single data table is determined to be less than a data threshold, then the transaction processing engine is selected as the target processing engine. That is, when a data processing request involves a single data table, and the data volume of the table is relatively small, the amount of data that the data processing request needs to process is also relatively small, so the data processing request is sent to the transaction processing engine.

[0188] Example 2: Data table information indicates processing of data on a single data table. Based on the data access columns in the condition information and the column definitions of the data access columns in the database environment information, it is determined that the data processing request is a single-table point-and-click processing request. Therefore, the transaction processing engine is selected as the target processing engine. When processing a single data table, if the condition information is a query condition for that single data table, and the query condition includes a data access column, the column definition of the access column is determined from the metadata of the database environment information. If the access column is determined to be a primary key, an enumeration type, or has a unique value constraint or an ordered index, the data processing request can be determined to be a single-table point-and-click request. If the amount of data to be processed in the data processing request is relatively small, the data processing request is sent to the transaction processing engine.

[0189] Example 3: Data table information indicates the processing of data on a single data table. Based on the data access column in the condition information and the statistical information indicating data distribution in the database environment information, if the data distribution on the data access column meets the distribution conditions, then the transaction processing engine is selected as the target processing engine. To ensure data processing efficiency, when selecting the target processing engine, after determining that a specific data column on a single data table will be processed, the data distribution on that column is further considered. If the data is evenly distributed and the amount of data to be processed in the data processing request is relatively small, then the transaction processing engine is suitable for point lookup using an index, and the data processing request is sent to the transaction processing engine.

[0190] Example 4: Data table information indicates that data on a single data table needs processing. Based on the data access column in the condition information, combined with the statistical information indicating data distribution in the database environment information, it is determined that the data distribution on the data access column does not meet the distribution conditions (i.e., the data skew is severe). Combined with the data volume of the data table in the database environment information, it is determined that the data volume of the single data table is less than the data threshold. Therefore, the transaction processing engine is selected as the target processing engine. That is, when a data processing request involves a single data table, if the data volume of the data table is relatively small, even if a full table scan is performed, the amount of data that the data processing request needs to process is relatively small, and the data processing request is then sent to the transaction processing engine.

[0191] Example 5: Data table information indicates processing of data on a single data table. Based on the data access column in the condition information, combined with the statistical information indicating data distribution in the database environment information, it is determined that the data distribution on the data access column does not meet the distribution conditions. Combined with the data volume of the data table in the database environment information, it is determined that the data volume of a single data table exceeds a data threshold. Therefore, the analysis and processing engine is selected as the target processing engine. If the data distribution does not meet the distribution conditions and the data volume of the data table is relatively large, it will affect the evaluation of different execution plans by the transaction optimization engine in the transaction processing engine. Therefore, using an analysis and processing engine that only performs full table scans will not have this problem.

[0192] In this embodiment of the application, the condition information also includes: data filtering conditions and types (such as equality, range, existence, etc.) on the data access column. Therefore, data filtering conditions and types can also be considered when selecting the target processing engine. For example, when in a range query and the data on the data column is not unique, the analysis processing engine is selected as the target processing engine.

[0193] In Example 6, if the data table information indicates that data on a single data table needs to be processed, and the data volume of each data table in the database environment information is combined with the data volume of the single data table, and it is determined that the data volume on the single data table is greater than the data threshold, and the condition information indicates that a full table scan should be performed, then the analysis and processing engine is selected as the target processing engine; if the data volume on the data table is relatively large, and the data processing request needs to process a relatively large amount of data, then the data processing request is sent to the analysis and processing engine.

[0194] Example 7: The data table information indicates that data on multiple data tables will be processed. The condition information records the connection conditions and types (such as inner join, outer join, semi join, etc.) between the multiple data tables involved in the data processing request. Combined with the data volume and data distribution information of the data tables in the database environment information, if it is determined that the connection conditions and types meet the index availability, then the transaction processing engine is selected as the target processing engine.

[0195] For example, when selecting a target processing engine by combining the data volume and data distribution information of multiple data tables, if there is a target data table with a data volume less than the data volume threshold among the multiple data tables, and the data distribution information indicates that the data on the target data table meets the distribution conditions, then the transaction processing engine is selected as the target processing engine.

[0196] Another example is that when selecting a target processing engine by combining the data volume and data distribution information of multiple data tables, if there is a target data table with a data volume less than the data volume threshold, but the data distribution information indicates that the data on the target data table does not meet the distribution conditions, then the analysis and processing engine is selected as the target processing engine.

[0197] In one possible implementation, the distribution condition includes at least a uniform data distribution.

[0198] Example 8: The data table information indicates data processing on multiple data tables, and the condition information includes the query conditions on each data table. Then, the analysis and processing engine is selected as the target processing engine; that is, multiple data tables are processed separately. At this time, the data processing request needs to process data on multiple data tables, and the amount of data to be processed is relatively large. Therefore, the data processing request is sent to the analysis and processing engine.

[0199] See Figure 5 , Figure 5 This application provides a schematic diagram of a target selection processing engine based on a requested target, as illustrated in an embodiment of the present application. Figure 5As can be seen from the diagram: When a user triggers a data processing request, the data processing system receives the request, parses it to obtain the target, and then reads the data operation type from the target. If the data operation type is a non-read-only query, the data processing request is sent to the transaction processing engine as an OLTP request, and the transaction processing engine becomes the target processing engine for processing the data processing request. If the data operation type is a read-only query, the system continues to read the data table processing information from the target. If, based on the data table information and database environment information, it is determined that the amount of data to be processed by the data processing request is less than the processing threshold, the data processing request is sent to the transaction processing engine as an OLTP request, and the transaction processing engine becomes the target processing engine for processing the data processing request. If it is determined that the amount of data to be processed by the data processing request is greater than the processing threshold, the data processing request is sent to the analytical processing engine as an OLAP request, and the analytical processing engine becomes the target processing engine for processing the data processing request.

[0200] By combining the request target with database environment information to select the target processing engine, various factors affecting data processing response time and their interrelationships can be comprehensively considered, improving the accuracy of target processing engine selection. This allows the target optimizer in the target processing engine to generate a better execution plan for the received data processing requests, further improving data processing efficiency.

[0201] Step S402: Generate an execution plan for the data processing request using the target optimizer in the target processing engine.

[0202] Step S403: The data processing request is processed according to the execution plan by the target executor in the target processing engine.

[0203] It should be noted that the specific implementation methods of steps S402 and S403 can be found in the description of the target optimizer and target executor in the above data processing system, and will not be repeated here.

[0204] In this application, based on the parsed request target, a target processing engine is selected from the transaction processing engine and the analysis processing engine. Then, the target optimizer within the target processing engine generates an execution plan for the data processing request. Finally, the target executor within the target processing engine processes the data processing request according to the execution plan. By intelligently selecting the target processing engine based on the request target, utilizing its optimizer to generate an execution plan, and then using its executor to process according to the execution plan, data processing requests can be processed accurately and efficiently. This approach not only improves execution efficiency but also optimizes resource utilization and reduces latency, thereby enhancing overall system performance and user experience. This ensures processing quality while accelerating response speed and enhancing system flexibility and scalability. Furthermore, in HTAP service scenarios, generating effective execution plans for each received data processing request ensures accurate and error-free execution of the received data processing requests, improving data processing efficiency and thus enhancing overall performance.

[0205] To ensure the accuracy of the target processing engine used for data processing and further improve data processing efficiency, in this embodiment, after selecting a target processing engine for the data processing request based on the request target, the target optimizer in the target processing engine will perform performance verification on the processing engine selected based on the data running status before generating an execution plan for the data processing request. Based on the obtained verification results, the final target processing engine to be used will be determined.

[0206] See Figure 6 , Figure 6 A flowchart of a second data processing method provided in this application embodiment includes the following steps:

[0207] Step S600: Parse the received data processing request and determine the corresponding request target.

[0208] Step S601: Based on the request target, select a processing engine for the data processing request; the processing engine selected based on the request target is either a transaction processing engine or an analysis processing engine.

[0209] Step S602: Based on the data running status, perform performance verification on the processing engine selected based on the request target and obtain the corresponding verification results; wherein, the verification results indicate whether the target execution performance of the processing engine selected based on the request target reaches the preset performance index.

[0210] The data operation status includes at least one of the following: data storage status, data synchronization status, and data resource status.

[0211] Data storage status representation: The size and proportion of free and available space in the cache resource pool, representing the storage data read from the underlying storage medium;

[0212] Data synchronization status indicates the degree of synchronization between the data modifications of the transaction processing engine and the data copy used by the analysis processing engine through the asynchronous synchronization mechanism. This includes the size of the incremental data difference between the columnar storage data copy used by the analysis processing engine and the data in the transaction processing engine, as well as the current synchronization speed.

[0213] Data resource status indicates the remaining amount of computing and memory resources.

[0214] In one possible implementation, the data runtime state includes the data storage state; based on the data runtime state, the performance of the processing engine selected based on the request target is verified to obtain the corresponding verification results, including:

[0215] The scanned data is determined by the processing engine that performs a full table scan on the data access column based on the requested target. The scanned data is then matched with the stored data to obtain the matching results. The matching results represent the hit rate of the scanned data in the stored data.

[0216] If the hit rate is not less than the preset hit threshold, a verification result is obtained indicating that the target execution performance has reached the preset performance index.

[0217] See Figure 7 , Figure 7 This is a schematic diagram illustrating the first method of performance verification for a processing engine based on request target selection, provided in an embodiment of this application. Taking the processing engine based on request target selection as an example, it shows... Figure 7 From this, we can know that:

[0218] The data processing system's cache resource pool (such as a memory cache pool) reads data from the underlying storage medium and maintains stored data that has been read from the underlying storage medium recently. Since accessing data in the cache resource pool is much faster than accessing the underlying storage medium during data processing, the classifier, based on the scanned data obtained by the analysis and processing engine performing a full table scan of the data access column, accesses the stored data in the cache resource pool to obtain the data access hit rate, and obtains the verification result based on the hit rate. The hit rate is determined by the degree of matching between the scanned data and the stored data (i.e., the amount of scanned data stored in the stored data).

[0219] If it is determined that the amount of scanned data stored in the stored data exceeds the threshold, that is, the hit rate of scanned data in the stored data exceeds the preset hit threshold (e.g., 80%), the full table scan operation of the analysis and processing engine can avoid frequent disk I / O. Then the target execution performance of the selected analysis and processing engine reaches the preset performance index, the analysis and processing engine can obtain a performance advantage, and obtain a verification result indicating that the target execution performance has reached the preset performance index.

[0220] If the amount of scanned data stored in the storage data does not exceed the threshold, the remaining available capacity in the cache resource pool (i.e., the current occupancy of the cache resource pool in the data processing system) is determined. Furthermore, the amount of data in the target data table and the statistical information are estimated to predict the memory overhead required for the analysis and processing engine to perform a full table scan of the columns accessed in the data processing request. For data processing requests involving multiple table joins, the intermediate results during execution are also estimated. Based on the memory overhead, intermediate results, and remaining available capacity, it is determined whether the target execution performance of the selected analysis and processing engine meets the preset performance indicators. For example, if the remaining available capacity is very limited, but the amount of data accessed in the table is large, it is impossible to meet the threshold for the amount of data stored for a full table scan. In other words, the hit rate of the scanned data in the stored data cannot exceed the preset hit threshold (e.g., 80%). This is detrimental to the analysis and processing engine's performance. The full table scan operation will generate a large amount of memory cache swapping and access to the underlying storage media, leading to severe memory contention and excessive IO resource consumption. Therefore, the target execution performance of the selected analysis and processing engine does not meet the preset performance indicators, and a verification result indicating that the target execution performance does not meet the preset performance indicators is obtained.

[0221] In one possible implementation, if the processing engine selected based on the request target is an analysis processing engine, the data operation status includes the data synchronization status; based on the data operation status, performance verification is performed on the processing engine selected based on the request target to obtain the corresponding verification results, including:

[0222] Based on the data synchronization status, if it is determined that the data modifications processed by the transaction processing engine are completely synchronized to the data copy used by the analysis processing engine, then a verification result indicating that the target execution performance has reached the preset performance index is obtained.

[0223] In the embodiments of this application, both the transaction processing engine and the analytical processing engine have their own applicable data storage formats. Typically, data processing systems asynchronously synchronize data modifications processed by the transaction processing engine to the data copy used by the analytical processing engine. See also... Figure 8 , Figure 8 This is a schematic diagram of data status synchronization processing provided in an embodiment of this application. Figure 8 As can be seen, the data processing system first caches the data modifications processed by the transaction processing engine in incremental storage, and then batch replays them into the data copy used by the analysis processing engine.

[0224] If the data replica used by the analytical processing engine is not fully synchronized with the data modifications processed by the transaction processing engine, the analytical processing engine must wait until all the data accessed by the incremental storage is fully synchronized to the data replica it uses before it can begin execution. This will increase the data processing request response time, and the impact will be even more severe if there is significant write pressure on the accessed data table. In this case, even if the analytical processing engine has superior processing capabilities compared to the transaction processing engine, a verification result indicating that the target execution performance has not met the preset performance indicators will be obtained. Conversely, if the data replica used by the analytical processing engine is fully synchronized with the data modifications processed by the transaction processing engine, a verification result indicating that the target execution performance meets the preset performance indicators will be obtained.

[0225] In one possible scenario, if the write load accounts for a high proportion of the data processing system load and a large amount of write load falls on the data tables, the pressure on the data replicas used by the analysis and processing engine to synchronize updated data will increase, the synchronization latency will increase, and the response time of the data processing request of the analysis and processing engine will be longer due to waiting for the synchronization of incremental data or processing a large amount of incremental data. In this case, a verification result indicating that the target execution performance has reached the preset performance index will be obtained, and it may be more appropriate to send it to the transaction processing engine for execution.

[0226] The data processing system load includes: computing resource usage, disk read pressure, and read / write volume of other concurrently executed queries on the tables and columns involved in this query.

[0227] In one possible implementation, the data runtime state includes the data resource state. Based on the data runtime state, the performance of the processing engine selected based on the request target is verified to obtain the corresponding verification results, including:

[0228] Estimate the amount of target resources required for the processing engine selected based on the request target to execute the data processing request. If the target resource amount is not greater than the remaining resource amount, a verification result indicating that the target execution performance has reached the preset performance index is obtained.

[0229] In one possible implementation, the remaining resources are the remaining computing and memory resources of the data processing system.

[0230] See Figure 9 , Figure 9 This is a schematic diagram illustrating a second method for performance verification of a processing engine based on request target selection, provided in an embodiment of this application. Taking the request target selection-based processing engine as an example, it shows... Figure 9 From this, we can know that:

[0231] High performance of the analysis and processing engine is accompanied by the consumption of a large amount of computing and memory resources. Therefore, we first estimate the target amount of resources required for the analysis and processing engine to process data processing requests, and then determine the amount of available remaining resources based on the current load status of the data processing system. Finally, we obtain the verification results based on the comparison between the remaining resources and the target amount of resources.

[0232] If the data processing system is under low load and has a large amount of remaining resources, which is greater than the target amount of resources, sufficient computing and memory resources can be allocated to the analysis and processing engine. At this time, a verification result indicating that the target execution performance has reached the preset performance index will be obtained.

[0233] If the data processing system is under high load and has limited remaining resources, to ensure that the response time of currently processed data processing requests and the ability to respond to subsequent data processing requests are not affected, sufficient computing and memory resources cannot be allocated to the analysis and processing engine, thus impacting its request processing time. In other words, the remaining resources are less than the target amount, limiting the performance of the analysis and processing engine. Insufficient available memory resources will increase the memory swap rate, failing to fully utilize computing resources and leading to performance degradation. Furthermore, continuing to select the analysis and processing engine under high load will result in insufficient concurrency leading to performance degradation, and will also exacerbate thread contention and thread switching. This will not only cause performance degradation for the current query but also affect the execution efficiency of other queries being processed. In this case, a verification result indicating that the target execution performance has not reached the preset performance indicators is obtained.

[0234] Step S603: Based on the verification results, determine whether the target execution performance has reached the preset performance index. If yes, proceed to step S604; otherwise, proceed to step S605.

[0235] Step S604: Based on the verification results, when the target execution performance reaches the preset performance index, determine to use the target processing engine selected based on the request target.

[0236] Step S605: Based on the verification results, if it is determined that the target execution performance has not reached the preset performance index, replace the processing engine selected based on the request target with another processing engine, and use the replaced processing engine as the target processing engine when processing the data processing request.

[0237] For example, the processing engine selected based on the request target is the analysis processing engine. After verifying the analysis processing engine based on the data running status, if it is determined that the target execution performance of the analysis processing engine has not reached the preset performance index, then the transaction processing engine is used to replace the analysis processing engine, and the transaction processing engine is used as the target processing engine when processing the data processing request, that is, the transaction processing engine is used to process the data processing request.

[0238] Step S606: Generate an execution plan for the data processing request using the target optimizer in the target processing engine.

[0239] Step S607: The data processing request is processed according to the execution plan by the target executor in the target processing engine.

[0240] In one possible implementation, when verifying the processing engine selected based on the request target based on the data running status, the execution performance of the analysis and processing engine can also be estimated based on the data storage status, data resource status, and data synchronization status to obtain the corresponding estimated execution performance. Based on the obtained three estimated execution performances, the target execution performance of the analysis and processing engine is comprehensively determined. Then, the verification result is determined according to the comparison relationship between the target execution performance and the preset performance indicators. Finally, based on the verification result, the target processing engine used in the data processing process is determined.

[0241] For example, when determining the target performance index of the analysis and processing engine based on the three estimated execution performances obtained, a weighted approach can be used.

[0242] For ease of understanding, this application provides a schematic diagram of a first method for determining the target processing engine used in data processing, see [link to schematic diagram]. Figure 10 Taking the processing engine selected based on the request target as an example, from... Figure 10 From this, we can know that:

[0243] After selecting the analysis and processing engine based on the request target, the classifier estimates the execution performance of the analysis and processing engine based on the data storage status, data resource status, and data synchronization status, obtaining corresponding estimated execution performance. The three estimated execution performances are then weighted to determine the target execution performance of the analysis and processing engine. This target execution performance is compared with preset performance indicators to determine the verification result. When the verification result indicates that the target execution performance exceeds the preset performance indicators, the analysis and processing engine is used for processing, i.e., the data processing request is sent to the analysis and processing engine as an OLAP request. When the verification result indicates that the target execution performance does not exceed the preset performance indicators, the transaction processing engine is used for processing, i.e., the data processing request is sent to the transaction processing engine as an OLTP request.

[0244] In this application, runtime states such as data storage status, data synchronization status, and data resource status are considered to further determine the target processing engine used during data processing, ensuring the performance of the selected target processing engine. Then, the target optimizer of the target processing engine generates the optimal execution plan, avoiding the problem of search space expansion when generating the execution plan, and effectively ensuring fast query response and system throughput under complex loads.

[0245] To ensure the accuracy of the target processing engine selected subsequently, in this embodiment of the application, after the target executor in the target processing engine finishes processing the data processing request according to the execution plan, the actual response time of the target processing engine is statistically analyzed and compared with the corresponding response time standard. If the actual response time is greater than the response time standard, the request target and the actual response time of this data processing request are statistically analyzed and stored in the anomaly database of the data processing system to assist in the analysis of subsequent data processing requests.

[0246] In one possible implementation, the response time standard includes, but is not limited to: a preset response time, or an estimated response time that the data processing system pre-calculates for a data processing request.

[0247] When the response time standard is the estimated response time, the estimated response time is determined by the data processing system through a target prediction model obtained through training, based on the data processing request, database environment information, and data operation status. The target prediction model is obtained by training the data processing request, database environment information, data operation status, and the correspondence between the actual response time and the data processing request.

[0248] In one possible implementation, when the request target and the actual response time are statistically analyzed as abnormal data, the database environment information and data operation status during this data processing can also be statistically analyzed and stored in the abnormal database in correspondence with the request target.

[0249] See Figure 11 , Figure 11 This is a schematic diagram illustrating the storage of historical anomaly data provided in an embodiment of this application, using the target processing engine as the analysis processing engine and a preset response time as the response time standard. Figure 11 From this, we can know that:

[0250] After the classifier determines that the target processing engine is the analytical processing engine, the data processing request is sent to the analytical processing engine as an OLAP request. Within the analytical processing engine, the analytical processing optimizer generates an execution plan, which is then executed by the analytical processing executor. The response time of the analytical processing engine is also monitored in real time. After the analytical processing engine finishes processing the data processing request associated with the data processing request, the actual response time is obtained and compared with a preset response time. If the actual response time exceeds the preset response time, the request target, database environment information, and the actual response time of the analytical processing engine are stored in an anomaly database. This database serves as historical anomaly data to assist in determining the execution engine for subsequent data processing requests. The historical anomaly data represents the correspondence between historical request targets, historical database environment information, historical processing engines, and historical actual response times.

[0251] In this embodiment, different preset response times can be set for different processing engines, and the actual response time can be compared with the matching preset response time. For example, an OLAP preset response time is set for the analytics processing engine, and an OLTP preset response time is set for the transaction processing engine. The actual response time of the analytics processing engine is compared with the OLAP preset response time, and the actual response time of the transaction processing engine is compared with the OLTP preset response time. If the actual response time is greater than the associated preset response time, abnormal data is stored, increasing the amount of actual response time data maintained in the abnormal data and improving the accuracy of the prediction.

[0252] In one possible implementation, a standard range for exceeding the response time standard can be set. When the actual response time exceeds the response time standard, but the percentage of exceeding the response time standard is within the standard range, the request target, database environment information, etc. of the current data processing request will not be stored as abnormal data.

[0253] In one possible implementation, the abnormal data stored in the abnormal database can be based on the Least Recently Used (LRU) strategy and employ a dynamic sliding window mechanism to maintain the correspondence between the request target, database environment information, database running status, and actual response time of the processing engine used for data processing requests in the most recent N hours (configurable time window). This ensures that the data processing system can update the abnormal data in a timely and dynamic manner, reduce the storage load of the data processing system, eliminate the impact of special cases of individual abnormalities, improve the accuracy of the data processing engine's verification process, and can quickly respond to scenarios with dynamic loads.

[0254] Based on storing historical abnormal data in the data processing system, in this embodiment of the application, after selecting a target processing engine for the data processing request based on the request target, the target optimizer in the target processing engine will perform performance verification on the processing engine selected based on the request target based on the historical abnormal data before generating an execution plan for the data processing request. Based on the obtained verification results, the final target processing engine to be used will be determined.

[0255] See Figure 12 , Figure 12 The flowchart of the third data processing method provided in this application embodiment includes the following steps:

[0256] Step S1200: Parse the received data processing request and determine the corresponding request target.

[0257] Step S1201: Based on the request target, select a processing engine for the data processing request; the processing engine selected based on the request target is either a transaction processing engine or an analysis processing engine.

[0258] Step S1202: Based on historical anomaly data, perform performance verification on the processing engine selected based on the request target and obtain the corresponding verification results; wherein, the verification results indicate whether the target execution performance of the processing engine selected based on the request target has reached the preset performance indicators; the historical anomaly data includes at least the correspondence between the request target of the historical processing request, the database environment information and the actual response time of the historical processing engine.

[0259] For example, when historical abnormal data exists, after selecting a processing engine based on the request target, before sending the data processing request to the selected processing engine, it is also necessary to identify historical processing engines that are consistent with the selected processing engine recorded in the historical database, and identify at least one historical abnormal data associated with the historical processing engine. The historical abnormal data includes: historical request target, historical database environment information, historical actual response time, etc.; determine the feature similarity between the request target of the data processing request and at least one historical request target, and select historical abnormal data with a feature similarity greater than a feature similarity threshold from at least one historical abnormal data based on at least one feature similarity; determine the environment similarity between the current database environment information and at least one historical database environment information, and select historical abnormal data with an environment similarity greater than an environment similarity threshold from at least one historical abnormal data based on at least one environment similarity; further determine the historical actual response time associated with the selected historical abnormal data, and calculate the average percentage of the selected historical actual response time exceeding the response time standard. When the average percentage exceeds the percentage threshold, a verification result is obtained that the target execution performance has not reached the preset performance index; similarly, when the average percentage is lower than the percentage threshold, a verification result is obtained that the target execution performance has reached the preset performance index.

[0260] In one possible implementation, additional rules can be configured to assist in the selection. For example, a high-priority rule: if the processing engine selected based on the request target is the analytics processing engine, and the actual response time of similar data processing requests recorded is significantly higher than the preset time standard while the synchronization latency overhead is similar, the transaction processing engine is directly switched to the transaction processing engine. A low-priority rule: accepts minor fluctuations in the analytics processing engine.

[0261] Step S1203: Based on the verification result, determine whether the target execution performance has reached the preset performance index. If yes, proceed to step S1204; otherwise, proceed to step S1205.

[0262] Step S1204: Based on the verification results, when it is determined that the target execution performance reaches the preset performance index, the target processing engine selected based on the request target is used.

[0263] Step S1205: Based on the verification results, if it is determined that the target execution performance has not reached the preset performance index, use other processing engines to replace the processing engine selected based on the request target, and use the replaced processing engine as the target processing engine when processing the data processing request.

[0264] Step S1206: Generate an execution plan for the data processing request through the target optimizer in the target processing engine.

[0265] Step S1207: The data processing request is processed according to the execution plan by the target executor in the target processing engine.

[0266] In one possible implementation, the processing engine used after data operation status verification can also be verified based on the abnormal data. In this case, the abnormal data stored in the abnormal database includes: the request target, database environment information, data operation status, and the correspondence between the actual response time and the processing engine.

[0267] In this application, before using the selected target processing engine for processing, the selected target processing engine is verified based on historical abnormal data to further ensure the performance of the processing engine used and improve data processing efficiency.

[0268] In this embodiment of the application, when selecting a target processing engine for a data processing request based on the request target, it can be executed through a target classifier, which is obtained by training on a data processing request labeled with the processor engine.

[0269] When selecting a target processing engine for a data processing request using a target classifier, factors such as the request target, database environment information, data processing status, and verification based on abnormal data can be comprehensively considered to determine the target processing engine to be used during the data processing process. See also Figure 13 , Figure 13 This is a schematic diagram illustrating the second method of determining the target processing engine used in data processing, as provided in the embodiments of this application. Figure 13 From this, we can know that:

[0270] The parser parses and processes the data processing request to obtain the corresponding request target. The request target contains the data operation type and data processing information, including but not limited to: data table information, query conditions, join conditions, and projection columns. The data table information records the total number of data tables involved in the data processing request and the total number of data tables to be accessed. The query conditions record the data filtering conditions and types on each data access column, such as equality, range, and existence. The join conditions record the join conditions and types between the multiple data tables involved in the data processing request, such as inner join, outer join, and semi-join. The projection columns record the type of result to be returned, such as constant, column value, or function calculation result on a column value.

[0271] Furthermore, based on the request objective and considering database environment information and data operation status, four key metrics were determined for the transaction processing engine and the analytical processing engine: point lookup, full table scan, contention level, and synchronization latency overhead. The point lookup and full table scan metrics characterize the suitability of the transaction processing engine and the analytical processing engine, with a higher point lookup value indicating a stronger preference for the transaction processing engine. The contention level characterizes the read / write pressure on the data used by the transaction processing engine and the analytical processing engine in relation to the current data processing request. The synchronization latency overhead metric is almost zero for the transaction processing engine and is primarily used to characterize the cost of processing incremental data for the analytical processing engine. Based on these four metrics, the execution cost of the transaction processing engine and the analytical processing engine was comprehensively determined, and the engine with the lower cost was selected as the chosen execution engine; the lower the execution cost, the better the execution performance.

[0272] Finally, the selected execution engine is validated using historical exception data stored in the exception database, and the final processing engine is determined based on the validation results.

[0273] This application proposes a method to estimate the execution costs of the transaction processing engine and analytical processing engine in HTAP based on the request target, database environment information, and data running status of the data processing request. By comparing the execution costs of the two processing engines, the engine with the lower execution cost is selected as the target processing engine. An execution plan is generated by the target optimizer in the target processing engine, and the target executor processes the data according to the execution plan. This method comprehensively considers various factors affecting data processing response time and their interrelationships, overcoming the deficiency of traditional optimizers that do not consider data running status. By leveraging the powerful reasoning capabilities of AI and combining real-time response data, it can greatly improve the ability to select the optimal execution plan, increase execution efficiency, and enhance throughput performance.

[0274] Based on the same inventive concept, embodiments of this application also provide a data processing apparatus. See also Figure 14 , Figure 14 A schematic diagram of the data processing device 1400 may include:

[0275] The determining unit 1401 is used to parse and process the received data processing request and determine the corresponding request target;

[0276] Selection unit 1402 is used to select a target processing engine for a data processing request based on the request target; the target processing engine is a transaction processing engine or an analysis processing engine.

[0277] The generation unit 1403 is used to generate an execution plan for a data processing request through the target optimizer in the target processing engine.

[0278] The processing unit 1404 is used to process data processing requests according to the execution plan through the target executor in the target processing engine;

[0279] When the target processing engine is a transaction processing engine, the target optimizer is a transaction processing optimizer, and the target executor is a transaction processing executor; when the target processing engine is an analysis processing engine, the target optimizer is an analysis processing optimizer, and the target executor is an analysis processing executor.

[0280] In one possible implementation, the request target includes: a data operation type; the selection unit 1402 is specifically used for:

[0281] If the data operation type indicates that the data processing request is a non-read-only query, then the transaction processing engine is selected as the target processing engine.

[0282] In one possible implementation, the request target includes: data operation type and data processing information; the selection unit 1402 is specifically used for:

[0283] If the data operation type indicates that the data processing request is a read-only query, then the target processing engine is selected for the data processing request based on the data processing information and the database environment information.

[0284] In one possible implementation, the data processing information includes data table information and condition information; the selection unit 1402 is specifically used for:

[0285] If the data table information indicates that data on a single data table needs to be processed, and based on the data access column in the condition information and the column definition of the data access column in the database environment information, it is determined that the data processing request is a single table lookup processing request, then the transaction processing engine is selected as the target processing engine.

[0286] If the data table information indicates that data on multiple data tables needs to be processed, and the condition information indicates that there is a connection relationship between the multiple data tables, then the target processing engine is selected for the data processing request based on the data volume and data distribution information of the data tables in the database environment information.

[0287] In one possible implementation, unit 1402 is specifically used for:

[0288] If there is a target data table with a data volume less than the data volume threshold among multiple data tables, and the data distribution information indicates that the data on the target data table meets the distribution conditions, then the transaction processing engine is selected as the target processing engine.

[0289] If there is a target data table with a data volume less than the data volume threshold among multiple data tables, but the data distribution information indicates that the data on the target data table does not meet the distribution conditions, then the analysis and processing engine is selected as the target processing engine.

[0290] In one possible implementation, after selecting a target processing engine for the data processing request based on the request target, and before generating an execution plan for the data processing request through the target optimizer in the target processing engine, the selection unit 1402 is further configured to:

[0291] Based on the data operation status, the performance of the processing engine selected based on the request target is verified, and the corresponding verification results are obtained. The verification results indicate whether the target execution performance of the processing engine selected based on the request target reaches the preset performance index.

[0292] Based on the verification results, when the target execution performance reaches the preset performance index, the target processing engine is determined to be used.

[0293] In one possible implementation, the data operation state includes at least one of the following: data storage state, data synchronization state, and data resource state;

[0294] Among them, data storage status refers to the stored data read from the underlying storage medium in the cache resource pool;

[0295] Data synchronization status indicates the degree of synchronization between the data modifications made by the transaction processing engine and the data replicas used by the analytical processing engine through an asynchronous synchronization mechanism.

[0296] Data resource status indicates the remaining amount of computing and memory resources.

[0297] In one possible implementation, if the processing engine selected based on the request target is an analysis processing engine, the data running state includes the data storage state; the selection unit 1402 is further configured to:

[0298] The analysis and processing engine identifies the scanned data from the data access columns that were scanned. This scanned data is then matched against the stored data to obtain the matching results. The matching results represent the hit rate of the scanned data in the stored data.

[0299] If the hit rate is not less than the preset hit threshold, a verification result is obtained indicating that the target execution performance has reached the preset performance index.

[0300] In one possible implementation, if the processing engine selected based on the request target is an analysis processing engine, the data running state includes a data synchronization state; the selection unit 1402 is further used for:

[0301] Based on the data synchronization status, if it is determined that the data modifications processed by the transaction processing engine are completely synchronized to the data copy used by the analysis processing engine, then a verification result indicating that the target execution performance has reached the preset performance index is obtained.

[0302] In one possible implementation, if the processing engine selected based on the request target is an analysis processing engine, the data running state includes the data resource state; the selection unit 1402 is further configured to:

[0303] The estimated analysis and processing engine calculates the target amount of resources required to process the data processing request. If the target amount of resources is not greater than the remaining resources, a verification result is obtained indicating that the target execution performance has reached the preset performance index.

[0304] In one possible implementation, the selection unit 1402 is further used for:

[0305] Based on the verification results, if the target execution performance does not meet the preset performance indicators, use other processing engines to replace the processing engine selected based on the request target, and use the replaced processing engine as the target processing engine when processing data processing requests.

[0306] In one possible implementation, after selecting a target processing engine for the data processing request based on the request target, and before generating an execution plan for the data processing request through the target optimizer in the target processing engine, the selection unit 1402 is further configured to:

[0307] Based on historical anomaly data, the performance of the processing engine selected based on the request target is verified, and the corresponding verification results are obtained. The verification results indicate whether the target execution performance of the processing engine selected based on the request target meets the preset performance indicators.

[0308] Based on the verification results, when the target execution performance reaches the preset performance index, the target processing engine is determined to be used.

[0309] Among them, historical anomaly data includes at least the correspondence between historical processing requests and the actual response time of the historical processing engine.

[0310] In one possible implementation, unit 1402 is selected for:

[0311] The target classifier is used to select a target processing engine for the data processing request based on the request target; the target classifier is obtained by training on the data processing request with processor engine annotations.

[0312] In HTAP service scenarios, a high-performance execution plan is generated for each received data processing request, ensuring that the received data processing requests are executed accurately and without error, improving data processing efficiency, and thus improving overall performance.

[0313] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0314] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0315] After introducing the data processing method and apparatus according to exemplary embodiments of this application, the electronic device according to another exemplary embodiment of this application will be described next.

[0316] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0317] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In one embodiment, the electronic device may be a server, such as... Figure 1 The server 120 is shown. In this embodiment, the structure of the electronic device can be as follows: Figure 15 As shown, it includes a memory 1501, a communication module 1503, and one or more processors 1502.

[0318] The memory 1501 is used to store computer programs executed by the processor 1502. The memory 1501 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0319] Memory 1501 may be volatile memory, such as random-access memory (RAM); memory 1501 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1501 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1501 may be a combination of the above-described memories.

[0320] Processor 1502 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 1502 is used to implement the above-described data processing method when calling computer programs stored in memory 1501.

[0321] The communication module 1503 is used to communicate with terminal devices and other servers.

[0322] This application embodiment does not limit the specific connection medium between the memory 1501, communication module 1503, and processor 1502. This application embodiment... Figure 15 The memory 1501 and the processor 1502 are connected via a bus 1504, and the bus 1504 is in Figure 15 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1504 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 15 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0323] In some possible implementations, memory 1501 stores a computer storage medium containing computer-executable instructions for implementing the steps of the data processing methods of the embodiments of this application. Processor 1502 is used to execute the aforementioned data processing methods.

[0324] In some possible implementations, various aspects of the data processing method provided in this application may also be implemented in the form of a program product, which includes a computer program that, when the program product is run on an electronic device, causes the electronic device to perform the steps of the data processing method according to the various exemplary embodiments of this application described above.

[0325] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0326] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0327] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0328] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0329] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's electronic device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0330] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0331] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0332] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method, characterized in that, The method includes: The received data processing requests are parsed and processed to determine the corresponding request target; Based on the requested objective, a target processing engine is selected; the target processing engine is either a transaction processing engine or an analysis processing engine. An execution plan is generated for the data processing request using the target optimizer in the target processing engine. The data processing request is processed according to the execution plan by the target executor in the target processing engine. Wherein, when the target processing engine is a transaction processing engine, the target optimizer is a transaction processing optimizer, and the target executor is a transaction processing executor; when the target processing engine is an analysis processing engine, the target optimizer is an analysis processing optimizer, and the target executor is an analysis processing executor.

2. The method as described in claim 1, characterized in that, The target of the request includes: data operation type; The step of selecting a target processing engine for the data processing request based on the request target includes: If the data operation type indicates that the data processing request is a non-read-only query, then the transaction processing engine is selected as the target processing engine.

3. The method as described in claim 1, characterized in that, The requested target includes: data operation type and data processing information; The step of selecting a target processing engine for the data processing request based on the request target includes: If the data operation type indicates that the data processing request is a read-only query, then based on the data processing information and the database environment information, a target processing engine is selected for the data processing request.

4. The method as described in claim 3, characterized in that, The data processing information includes data table information and condition information; The step of selecting a target processing engine for the data processing request based on the data processing information and database environment information includes: If the data table information indicates that data on a single data table is to be processed, and based on the data access column in the condition information and combined with the column definition of the data access column in the database environment information, it is determined that the data processing request is a single table lookup processing request, then the transaction processing engine is selected as the target processing engine. If the data table information indicates that data on multiple data tables will be processed, and the condition information indicates that there is a connection relationship between the multiple data tables, then the target processing engine is selected for the data processing request by combining the data volume and data distribution information of the data tables in the database environment information.

5. The method as described in claim 4, characterized in that, The step of selecting a target processing engine for the data processing request by combining the data volume and data distribution information of the data tables in the database environment information includes: If there is a target data table in the plurality of data tables whose data volume is less than the data volume threshold, and the data distribution information indicates that the data on the target data table meets the distribution conditions, then the transaction processing engine is selected as the target processing engine. If there is a target data table in the plurality of data tables whose data volume is less than the data volume threshold, and the data distribution information indicates that the data on the target data table does not meet the distribution conditions, then the analysis and processing engine is selected as the target processing engine.

6. The method according to any one of claims 1-5, characterized in that, After selecting a target processing engine for the data processing request based on the request target, and before generating an execution plan for the data processing request through the target optimizer in the target processing engine, the method further includes: Based on the data running status, the performance of the processing engine selected based on the request target is verified to obtain the corresponding verification results; wherein, the verification results indicate whether the target execution performance of the processing engine selected based on the request target reaches the preset performance index. Based on the verification results, when the target execution performance reaches the preset performance index, the target processing engine is determined to be used.

7. The method as described in claim 6, characterized in that, The data operation status includes at least one of the following: data storage status, data synchronization status, and data resource status; The data storage status refers to the stored data read from the underlying storage medium in the cache resource pool; The data synchronization status indicates the degree of synchronization between the data modifications made by the transaction processing engine and the data replicas used by the analysis processing engine through an asynchronous synchronization mechanism. The data resource status refers to the remaining amount of computing resources and memory resources.

8. The method as described in claim 7, characterized in that, If the processing engine selected based on the requested target is an analysis processing engine, the data running status includes the data storage status; The process of performing performance verification on the processing engine selected based on the request target based on the data operation status, and obtaining the corresponding verification results, includes: The analysis and processing engine determines the scanned data from the full table scan of the data access column, matches the scanned data with the stored data, and obtains a matching result; wherein, the matching result represents the hit rate of the scanned data in the stored data; If the hit rate is not less than the preset hit threshold, a verification result is obtained indicating that the target execution performance has reached the preset performance index.

9. The method as described in claim 7, characterized in that, If the processing engine selected based on the requested target is an analysis processing engine, the data running status includes the data synchronization status; The process of performing performance verification on the processing engine selected based on the request target based on the data operation status, and obtaining the corresponding verification results, includes: Based on the data synchronization status, if it is determined that the data modifications processed by the transaction processing engine are completely synchronized to the data copy used by the analysis processing engine, then a verification result indicating that the target execution performance has reached the preset performance index is obtained.

10. The method as described in claim 7, characterized in that, If the processing engine selected based on the requested target is an analysis processing engine, the data running status includes the data resource status; The process of performing performance verification on the processing engine selected based on the request target based on the data operation status, and obtaining the corresponding verification results, includes: Estimate the target amount of resources required by the analysis and processing engine to perform processing for the data processing request. If the target amount of resources is not greater than the remaining amount of resources, obtain a verification result indicating that the target execution performance has reached a preset performance index.

11. The method as described in claim 6, characterized in that, The method further includes: Based on the verification results, if it is determined that the target execution performance does not reach the preset performance index, another processing engine is used to replace the processing engine selected based on the request target, and the replaced processing engine is used as the target processing engine when processing the data processing request.

12. The method according to any one of claims 1-5, characterized in that, After selecting a target processing engine for the data processing request based on the request target, and before generating an execution plan for the data processing request through the target optimizer in the target processing engine, the method further includes: Based on historical anomaly data, the performance of the processing engine selected based on the request target is verified to obtain the corresponding verification result; wherein, the verification result indicates whether the target execution performance of the processing engine selected based on the request target reaches the preset performance index. Based on the verification results, when the target execution performance reaches the preset performance index, the target processing engine is determined to be used. The historical anomaly data includes at least the correspondence between historical processing requests and the actual response time of the historical processing engine.

13. The method according to any one of claims 1-5, characterized in that, The step of selecting a target processing engine for the data processing request based on the request target includes: Using a target classifier, a target processing engine is selected for the data processing request based on the requested target; wherein, the target classifier is obtained by training on data processing requests labeled with processor engines.

14. A data processing system, characterized in that, The system includes: a parser, a classifier, a transaction processing engine, and an analysis processing engine; the transaction processing engine includes: a transaction processing optimizer and a transaction processing executor; the analysis processing engine includes: an analysis processing optimizer and an analysis processing executor; wherein: The parser is used to: parse and process received data processing requests to determine the corresponding request target; The classifier is used to: select a target processing engine based on the request target; wherein the target processing engine is the transaction processing engine or the analysis processing engine; The target optimizer in the target processing engine is used to: generate an execution plan for the data processing request; The target executor in the target processing engine is used to: process the data processing request according to the execution plan; When the target processing engine is a transaction processing engine, the target optimizer is a transaction processing optimizer, and the target executor is a transaction processing executor; when the target processing engine is an analysis processing engine, the target optimizer is an analysis processing optimizer, and the target executor is an analysis processing executor.

15. A data processing apparatus, characterized in that, The device includes: The determining unit is used to parse and process the received data processing request and determine the corresponding request target; The selection unit is used to select a target processing engine based on the request target; the target processing engine is a transaction processing engine or an analysis processing engine. The generation unit is used to generate an execution plan for the data processing request through the target optimizer in the target processing engine; The processing unit is configured to process the data processing request according to the execution plan through the target executor in the target processing engine; Wherein, when the target processing engine is a transaction processing engine, the target optimizer is a transaction processing optimizer, and the target executor is a transaction processing executor; when the target processing engine is an analysis processing engine, the target optimizer is an analysis processing optimizer, and the target executor is an analysis processing executor.

16. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods described in claims 1-13.

17. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1-13.

18. A computer program product, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the methods described in claims 1-13.

Images