Data processing method and device, electronic equipment and computer readable storage medium

Abstract

The application relates to the technical field of data processing, is applied to the financial science and technology and intelligent medical treatment scenes, and provides a data processing method and device, an electronic equipment and a computer readable storage medium, the method comprises the following steps: acquiring a query statement; performing analysis processing on the query statement to obtain statement analysis information; performing transformation processing on the statement analysis information according to a query rewriting strategy to obtain query transformation information; constructing a query execution cost model based on data distribution statistical information corresponding to the query transformation information; performing execution cost evaluation processing according to the query transformation information and the query execution cost model to obtain a plurality of execution plan evaluation values; taking a query execution plan corresponding to the smallest execution plan evaluation value as a target query execution plan, and performing data query processing based on the target query execution plan. Through the technical scheme, the data query response time is reduced, and the data query efficiency is improved.

Description

Technical Field

[0001] The embodiments of this application relate to, but are not limited to, the field of data processing technology, and are applied to financial technology and smart healthcare scenarios. In particular, they relate to a data processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the continuous development of science and technology, various data management systems are increasingly being applied to people's daily lives. For example, in the field of fintech, loan data management systems can be used to manage and process various loan data, and transaction data management systems can be used to manage and process various transaction data. Similarly, in the field of smart healthcare, medical resource management systems can be used to manage and process various medical resource data, and inpatient management systems can be used to manage and process various inpatient-related data. However, as the amount of system data continues to increase, the query response time of the data management system becomes too long when processing complex queries, and the efficiency of data querying decreases. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0004] To address the problems mentioned in the background section, embodiments of this application provide a data processing method, apparatus, electronic device, and computer-readable storage medium that can reduce query complexity, significantly reduce data query response time, and improve data query efficiency.

[0005] In a first aspect, embodiments of this application provide a data processing method, the data processing method comprising: Retrieve the query statement; The query statement is parsed to obtain statement parsing information; The parsed information of the statement is transformed according to a preset query rewriting strategy to obtain query transformation information; A query execution cost model is constructed based on the data distribution statistics corresponding to the query transformation information; Based on the query transformation information and the query execution cost model, an execution cost evaluation process is performed to obtain multiple execution plan evaluation values; The query execution plan corresponding to the smallest execution plan evaluation value is taken as the target query execution plan, and data query processing is performed based on the target query execution plan.

[0006] Secondly, embodiments of this application also provide a data processing apparatus, the data processing apparatus comprising: The retrieval unit is used to retrieve the query statement; The parsing unit is used to parse the query statement to obtain statement parsing information; The transformation unit is used to transform the parsed information of the statement according to a preset query rewriting strategy to obtain query transformation information; The construction unit is used to construct a query execution cost model based on the data distribution statistics corresponding to the query transformation information; The evaluation unit is used to perform execution cost evaluation processing based on the query transformation information and the query execution cost model to obtain multiple execution plan evaluation values; The execution unit is used to take the query execution plan corresponding to the smallest execution plan evaluation value as the target query execution plan, and to perform data query processing based on the target query execution plan.

[0007] Thirdly, embodiments of this application also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data processing method described in the first aspect above.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for performing the data processing method described in the first aspect above.

[0009] The data processing method according to the embodiments provided in this application has at least the following beneficial effects: In the data processing process, a query statement is first obtained; then, the query statement is parsed to obtain statement parsing information; next, the statement parsing information is transformed according to a preset query rewriting strategy to obtain query transformation information; then, a query execution cost model is constructed based on the data distribution statistics corresponding to the query transformation information; next, execution cost evaluation is performed based on the query transformation information and the query execution cost model to obtain multiple execution plan evaluation values; finally, the query execution plan corresponding to the smallest execution plan evaluation value can be used as the target query execution plan, and data query processing is performed based on the target query execution plan. Through the above technical solution, transforming the statement parsing information according to the query rewriting strategy and performing execution cost evaluation using the query execution cost model can reduce query complexity, significantly reduce data query response time, and improve data query efficiency. Attached Figure Description

[0010] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0011] Figure 1 This is a schematic diagram of an application environment for a data processing method according to an embodiment of this application; Figure 2 This is a schematic flowchart of a data processing method provided in one embodiment of this application; Figure 3 yes Figure 2 A schematic diagram of a specific implementation method of step S200; Figure 4 yes Figure 2 A schematic diagram of a specific implementation of step S300; Figure 5 yes Figure 2 A schematic diagram of a specific implementation of step S400; Figure 6 yes Figure 2 A schematic diagram of a specific implementation of step S500; Figure 7 yes Figure 2 A schematic diagram of a specific implementation method of step S600; Figure 8 This is a flowchart illustrating a specific implementation method of the database data storage process; Figure 9 This is a schematic diagram of a data processing apparatus provided in one embodiment of this application; Figure 10 This is a schematic diagram of an electronic device provided in one embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0013] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0014] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0015] AI is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. Artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. Artificial intelligence can simulate the information processes of human consciousness and thought. Furthermore, artificial intelligence utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results—the theories, methods, technologies, and application systems available for use.

[0016] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0017] Artificial intelligence, or AI, is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0018] The servers involved in artificial intelligence technology can be standalone servers or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0019] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0020] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can receive query statements from the client; then parse the query statements to obtain statement parsing information; next, transform the statement parsing information according to a preset query rewriting strategy to obtain query transformation information; then, construct a query execution cost model based on the data distribution statistics corresponding to the query transformation information; next, perform execution cost evaluation based on the query transformation information and the query execution cost model to obtain multiple execution plan evaluation values; finally, the query execution plan corresponding to the smallest execution plan evaluation value can be used as the target query execution plan, and data query processing can be performed based on the target query execution plan. Through the above technical solution, transforming the statement parsing information according to the query rewriting strategy and using the query execution cost model for execution cost evaluation can reduce query complexity, significantly reduce data query response time, and improve data query efficiency. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description of specific embodiments of this application is provided below.

[0021] like Figure 2 As shown, Figure 2 This is a flowchart illustrating a data processing method provided in one embodiment of this application. The data processing method includes the following steps: Step S100: Obtain the query statement.

[0022] The data processing method provided in this application first obtains a query statement during the data processing process to prepare for subsequent data queries. For example, in the fintech field, when a user needs to query financial products with high current investment returns, they can input relevant query text, and the client can then convert the input query text into a corresponding query statement. Similarly, when a user needs to inquire about the distribution of bank branches, they can also input relevant query text, and the client can also convert the input query text into a corresponding query statement. In the smart healthcare field, when a user needs to query the distribution of antivenom serum in a region, they can input relevant query text, and the client can then convert the input query text into a corresponding query statement. Similarly, when a user needs to query the distribution of authoritative internal medicine doctors in a region, they can also input relevant query text, and the client can also convert the input query text into a corresponding query statement.

[0023] It is worth noting that during the process of obtaining query statements, when it involves processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when this application embodiment needs to obtain sensitive personal information of the user, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after explicitly obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of this application embodiment obtained.

[0024] Step S200: Parse the query statement to obtain statement parsing information.

[0025] The data processing method provided in this application, after obtaining a query statement, can parse the query statement to obtain statement parsing information, preparing for subsequent data queries. Specifically, in the process of parsing the query statement to obtain statement parsing information, the query statement can first undergo syntax parsing to obtain an abstract syntax tree; then, the abstract syntax tree obtained from the syntax parsing can undergo semantic parsing to obtain the corresponding statement parsing information.

[0026] It is worth noting that during the parsing process of a query statement, both syntax parsing and semantic parsing are required. Syntax parsing checks whether the query statement conforms to the syntax rules. Semantic parsing, based on correct syntax parsing, combines database metadata to verify whether the object exists, whether the type matches, and whether the permissions are sufficient.

[0027] like Figure 3 As shown, parsing a query statement to obtain statement parsing information can include the following steps: Step S210: Perform syntax parsing on the query statement to obtain an abstract syntax tree; Step S220: Perform semantic parsing on the abstract syntax tree to obtain statement parsing information.

[0028] For steps S210 to S220, in the process of parsing the query statement to obtain statement parsing information, the query statement is first parsed to obtain an abstract syntax tree; then, the abstract syntax tree is semantically parsed to obtain statement parsing information. Obtaining the statement parsing information provides a data foundation for subsequent transformations of the statement parsing information.

[0029] For example, in the field of smart healthcare, when it's necessary to extract key information, such as diagnosis and surgery names, from unstructured medical record text and fill it into a standard form, the backend checks the input items as a query statement. For instance, the query statement might be "Check if the discharge summary contains the pathology report number." Upon receiving this query statement, it can undergo syntactic parsing to obtain the corresponding abstract syntax tree (AST). Then, semantic parsing is performed on the AST to obtain the corresponding parsed information. Similarly, in the field of fintech, when a user needs to ask a smart customer service representative "How much was your credit card spending last month?", this is converted into a query statement. The query statement can then undergo syntactic parsing to obtain the corresponding AST, followed by semantic parsing to obtain the corresponding parsed information.

[0030] Step S300: Transform the statement parsing information according to the preset query rewriting strategy to obtain the query transformation information.

[0031] The data processing method provided in this application, after parsing the query statement to obtain the statement parsing information, can transform the corresponding statement parsing information according to a pre-set query rewriting strategy to obtain the corresponding query transformation information. Performing equivalent transformation processing on the statement parsing information can greatly simplify the statement parsing information and rewrite it into a more efficient execution form, making subsequent data queries easier, reducing many unnecessary calculation steps, and thus greatly improving the efficiency of data queries and saving data query response time.

[0032] like Figure 4 As shown, transforming the parsed information of a query according to a preset query rewriting strategy to obtain transformed query information may include the following steps: Step S310: Convert the statement parsing information into a statement relational algebra expression tree; Step S320: Normalize the predicates in the statement relational algebra expression tree to obtain the statement conjunction normal form; Step S330: Based on the predefined equivalent transformation rule base, perform equivalent transformation processing on the statement relational algebra expression tree that converts predicates into statement conjunction normal form to obtain query transformation information.

[0033] For steps S310 to S330, during the process of transforming the statement parsing information according to the pre-set query rewriting strategy to obtain query transformation information, the statement parsing information can be converted into a statement relational algebra expression tree; then, the predicates in the statement relational algebra expression tree are normalized to obtain the statement conjunctive normal form; then, based on the pre-set equivalence transformation rule base, the statement relational algebra expression tree converted from the predicates to the statement conjunctive normal form is subjected to equivalence transformation to obtain the corresponding query transformation information; through the above technical solution, the statement parsing information is rewritten into a more efficient execution form, making subsequent data queries simpler and reducing many unnecessary calculation steps.

[0034] In some embodiments of this application, the statement parsing information is converted into a statement relational algebra expression tree to prepare for subsequent normalization representation processing. The statement relational algebra expression tree is a tree-like data structure used internally by the data query optimizer to represent the logical execution process of a query statement at the relational algebra level.

[0035] It is worth noting that the predicates in the statement relational algebra expression tree include selection predicates and join predicates. These predicates can be expressions composed of fields, constants, comparison operators, and logical operators, defining the filtering or association logic of the data. Furthermore, standardizing the predicates in the statement relational algebra expression tree transforms the potentially chaotic and nested conditions in the original query information into a clear, analytical, and transformable standard form, laying the foundation for subsequent logical and physical optimizations.

[0036] It is worth noting that the equivalent transformation rule base in this application embodiment includes various data execution information, and the statement relational algebra expression tree includes multiple logical execution information. Therefore, based on the equivalent transformation rule base, the statement relational algebra expression tree that converts predicates into statement conjunctive normal form can be subjected to equivalent transformation processing to obtain query transformation information, which can greatly simplify the statement parsing information and rewrite it into a more efficient execution form, making subsequent data queries more convenient and reducing many unnecessary calculation steps.

[0037] For example, in the field of smart healthcare, when a medical big data research team needs to filter out complex cases from a large amount of patient data, after obtaining the corresponding statement parsing information, they can convert the statement parsing information into a statement relational algebra expression tree. Then, they can perform normalization processing on the predicates of the statement relational algebra expression tree to obtain the statement conjunctive normal form. Finally, based on a pre-defined equivalent transformation rule base, they can perform equivalent transformation processing on the statement relational algebra expression tree that has been converted from the predicates into the statement conjunctive normal form to obtain the query transformation information in the field of smart healthcare. Alternatively, in the fintech field, when banks need to perform credit scoring on customers' multidimensional data, the approval condition "rejecting applicants with more than 4 credit inquiries and a monthly income below 15,000" can be converted into a query statement. Subsequently, after obtaining the corresponding statement parsing information, this information can be converted into a statement relational algebra expression tree. Then, the predicates of the statement relational algebra expression tree can be normalized to obtain the statement conjunctive normal form. Finally, based on a pre-defined equivalent transformation rule base, the statement relational algebra expression tree, which has been converted from the predicates to the statement conjunctive normal form, can be subjected to equivalent transformation processing to obtain the query transformation information in the fintech field.

[0038] Step S400: Construct a query execution cost model based on the data distribution statistics corresponding to the query transformation information.

[0039] The data processing method provided in this application, after transforming the parsed information of the statement according to a preset query rewriting strategy to obtain query transformation information, can construct a query execution cost model based on the data distribution statistics corresponding to the query transformation information. Based on the query execution cost model, cost evaluation processing can be performed on the subsequent query execution plan to select the query execution plan with the smallest execution plan evaluation value, making the subsequent data query execution process simpler and the entire execution process simplified, greatly improving the efficiency of data query and saving data effect time.

[0040] It is worth noting that in the process of constructing the query execution cost model based on the data distribution statistics corresponding to the query transformation information, the corresponding data distribution statistics are first obtained from the pre-defined system metadata according to the table and column information of the query transformation information. Then, the corresponding query execution cost model can be constructed according to the data distribution statistics, pre-defined hardware resources and node status. Through the above technical solution, the generated query execution cost model can more accurately and reasonably evaluate each query execution plan, so as to select the query execution plan with the smallest execution plan evaluation value, and further simplify the data query process.

[0041] like Figure 5 As shown, constructing a query execution cost model based on the data distribution statistics corresponding to the query transformation information may include the following steps: Step S410: Based on the table and column information of the query transformation information, obtain the corresponding data distribution statistics from the preset system metadata; Step S420: Based on the data distribution statistics, preset hardware resources, and node status, a query execution cost model is constructed.

[0042] For steps S410 to S420, in the process of constructing the query execution cost model based on the data distribution statistics corresponding to the query transformation information, the corresponding data distribution statistics are first obtained from the pre-set system metadata according to the table and column information of the query transformation information; then, the corresponding query execution cost model can be constructed according to the data distribution statistics, pre-set hardware resources and node status; through the above technical solution, the generated query execution cost model can more accurately evaluate each query execution plan to select the query execution plan with the smallest execution plan evaluation value, which can further simplify the data query process and reduce the data query response time.

[0043] It is worth noting that the table information in the query transformation information consists of metadata details for the base tables, views, derived tables, or aliases of subqueries involved in the logical plan. Table information may include the table's unique identifier, table type (regular table, partitioned table, external table, temporary table, etc.), physical attributes (storage engine, tablespace, partition key, etc.), and statistical information references. The column information in the query transformation information consists of metadata details for columns referenced or output in the logical plan. Column information may include the table to which the column belongs and the column name, column data type, column constraints (whether it has a default value, whether it is a primary key / foreign key), column sorting rules, and column statistical information. The system metadata can be a data dictionary or a statistical information table; corresponding data distribution statistics (such as the number of rows in the table, the number of unique values ​​in the column, and a histogram of numerical distribution) are retrieved from the system metadata for subsequent cost estimation and physical execution plan selection.

[0044] It's worth noting that hardware resources can include the interface performance of memory and disk; node status can include node load and network latency. Subsequently, a corresponding query execution cost model can be constructed based on data distribution statistics, pre-defined hardware resources, and node status.

[0045] Step S500: Perform execution cost evaluation processing based on query transformation information and query execution cost model to obtain multiple execution plan evaluation values.

[0046] The data processing method provided in this application, after constructing a query execution cost model based on the data distribution statistics corresponding to the query transformation information, can perform execution cost evaluation processing based on the query transformation information and the query execution cost model to obtain multiple execution plan evaluation values, providing a basis for the selection of subsequent target query execution plans.

[0047] It is worth noting that in the process of obtaining multiple execution plan evaluation values ​​by evaluating the execution cost based on query transformation information and query execution cost model, multiple logical operations are first identified from the query transformation information; then, multiple candidate physical operations are identified based on the multiple logical operations and a pre-defined operation mapping library; next, the multiple candidate physical operations are enumerated and joined to obtain multiple query execution plans; finally, the multiple query execution plans are evaluated based on the query execution cost model to obtain multiple execution plan evaluation values. This provides a basis for subsequent selection of target query execution plans, thereby further simplifying the data query process, improving data query efficiency, and saving data query response time.

[0048] like Figure 6 As shown, the execution cost evaluation process, based on query transformation information and query execution cost model, yields multiple execution plan evaluation values, which may include the following steps: Step S510: Determine multiple logical operations from the query transformation information; Step S520: Based on multiple logical operations and a preset operation mapping library, multiple candidate physical operations are determined; Step S530: Enumerate and join multiple candidate physical operations to obtain multiple query execution plans; Step S540: Evaluate multiple query execution plans based on the query execution cost model to obtain multiple execution plan evaluation values.

[0049] For steps S510 to S540, in the process of obtaining multiple execution plan evaluation values ​​by evaluating the execution cost based on the query transformation information and the query execution cost model, firstly, multiple logical operations are determined from the query transformation information; then, multiple candidate physical operations are determined based on the multiple logical operations and a preset operation mapping library; next, the multiple candidate physical operations are enumerated and joined to obtain multiple query execution plans; finally, the multiple query execution plans can be evaluated based on the query execution cost model to obtain multiple execution plan evaluation values. This technical solution provides a basis for subsequent target query execution plan selection, thereby further simplifying the data query process, improving data query efficiency, and saving data query response time.

[0050] It is worth noting that multiple logical operations can be identified from the query transformation information, and the operation mapping library includes multiple physical operations, with each physical operation corresponding to a logical operation. Therefore, multiple candidate physical operations can be identified based on the obtained logical operations and the preset operation mapping library. Then, the multiple candidate physical operations can be enumerated and joined to obtain multiple query execution plans. Finally, the multiple query execution plans can be evaluated based on the query execution cost model to obtain multiple execution plan evaluation values, providing a basis for the selection of the target query execution plan in the future.

[0051] For example, in the field of smart healthcare, when medical staff need to query the blood routine reports of all patients in a certain department, they input the query text "Query the blood routine reports of all patients." After obtaining the corresponding query transformation information, multiple logical operations can be determined from the query transformation information. Then, based on the multiple logical operations and a pre-set manipulation mapping library, multiple candidate physical operations are determined. Next, the multiple candidate physical operations are enumerated and joined to obtain multiple query execution plans. Finally, the multiple query execution plans can be evaluated based on a query execution cost model to obtain multiple execution plan evaluation values. Alternatively, in the field of fintech, when business personnel need to query the average return of products in the previous quarter, they input the query text "Query the average return of all financial products under our company in the previous quarter." After obtaining the corresponding query transformation information, multiple logical operations can be determined from the query transformation information. Then, based on the multiple logical operations and a pre-set manipulation mapping library, multiple candidate physical operations are determined. Next, the multiple candidate physical operations are enumerated and joined to obtain multiple query execution plans. Finally, the multiple query execution plans can be evaluated based on a query execution cost model to obtain multiple execution plan evaluation values.

[0052] Step S600: Take the query execution plan corresponding to the smallest execution plan evaluation value as the target query execution plan, and perform data query processing based on the target query execution plan.

[0053] The data processing method provided in this application, after performing execution cost evaluation based on query transformation information and query execution cost model to obtain multiple execution plan evaluation values, can select the query execution plan corresponding to the smallest execution plan evaluation value as the target query execution plan, and perform data query processing based on the target query execution plan. Through the above technical solution, the data query process is further simplified, the efficiency of data query is improved, and the response time of data query is saved.

[0054] It is worth noting that in the process of data query processing based on the target query execution plan, the query execution steps and multiple execution sub-tasks are first determined according to the target query execution plan; then, the multiple execution sub-tasks can be executed according to the query execution steps to realize data query processing on the pre-set database, so as to achieve more convenient and efficient data query processing.

[0055] like Figure 7 As shown, data query processing based on the target query execution plan may include the following steps: Step S610: Determine the query execution steps and multiple execution subtasks based on the target query execution plan; Step S620: Perform execution processing on multiple execution subtasks according to the query execution steps to achieve data query processing on the preset database.

[0056] For steps S610 to S620, during the data query processing based on the target query execution plan, the target query execution plan first determines the query execution steps and multiple execution subtasks; then, the multiple execution subtasks can be executed according to the query execution steps to realize data query processing on the pre-set database, thereby achieving simpler and more efficient data query processing, reducing the response time of data query, and improving the efficiency of data query.

[0057] It is worth noting that after determining the query execution steps and multiple execution subtasks based on the target query execution plan, the obtained multiple execution subtasks can be executed according to the query execution steps. This allows for data query processing from the pre-defined database, making the entire data query process simpler and faster. Data query processing is performed based on the query execution plan with the lowest execution plan evaluation value, which simplifies the data query process, reduces the data query response time, and improves the efficiency of data query.

[0058] like Figure 8 As shown, the database is a distributed database, and the data storage process of the database may include the following steps: Step S710: Receive raw data; Step S720: Perform data sharding based on the data characteristics of the original data to obtain multiple data shards; Step S730: Perform load analysis on multiple nodes corresponding to the distributed database to obtain node load status information; Step S740: Distribute and store multiple sharded data according to the node load status information.

[0059] For steps S710 to S740, when the database is a distributed database, during the database data storage process, the original data can be received first; then, data sharding is performed based on the data characteristics of the original data to obtain multiple shards; next, load analysis is performed on the multiple nodes corresponding to the distributed database to obtain node load status information; finally, the multiple shards can be distributed and stored based on the node load status information. Through the above technical solution, distributed data storage can be achieved using load balancing, enabling more efficient data query processing in the future.

[0060] In some specific embodiments of this application, the business system or data acquisition terminal inputs raw data (such as business transaction records, user behavior logs, etc.) into the optimized data analysis system, triggering the data storage process. The system, based on data characteristics (data volume, business classification, query hotspots) and cluster resources, employs an efficient data sharding algorithm. For example, data is sharded according to time range and business dimensions to ensure balanced shard sizes, adaptability to query patterns, reduced invalid cross-shard communication, and improved read / write efficiency. For the sharded data, replicas are created on different nodes in the cluster according to an optimized replication strategy. By considering data importance and access frequency, a reasonable number and distribution of replicas are set (e.g., multiple replicas for hot data to improve read concurrency; simplified replicas for cold data to save storage), ensuring data reliability while balancing storage costs and access performance. The system monitors cluster node load (CPU utilization, memory usage, disk I / O activity) and network transmission status (data synchronization latency, inter-node communication bandwidth usage) in real time. When uneven node load is detected (e.g., some nodes have high CPU load for extended periods while others are idle), a dynamic load balancing mechanism is triggered to adjust the read / write scheduling of data shards, rationally allocating tasks to low-load nodes to ensure efficient resource utilization and avoid performance bottlenecks. Furthermore, when data enters the system, it is classified and marked according to preset rules (e.g., data sensitivity level configuration, field content characteristics), distinguishing sensitive data (e.g., user ID numbers, bank card information) from ordinary data to prepare for subsequent security processing. For data marked as sensitive, an encryption algorithm is used to encrypt the data before writing it to distributed storage, converting it into ciphertext for storage, ensuring that even if the data file is illegally accessed, the content cannot be directly parsed. During network transmission processes such as data synchronization between cluster nodes and return of query results to users, encryption protocols are enabled to encapsulate and encrypt data, preventing interception and tampering during transmission. When a user or application initiates a data access request (read, write, modify, etc.), the system verifies the requester's identity and permissions based on pre-configured fine-grained access control policies (such as user role permissions and data field permissions). Only authorized users are allowed to access the corresponding sensitive data, and unauthorized access attempts are rejected to ensure data access security. The system initiates a data backup process according to preset cycles (such as daily or weekly) or events (after batch data updates), backing up encrypted data (including metadata and business data) to secure storage media (such as off-site disaster recovery servers or object storage). When data loss or corruption is detected (such as disk failure or accidental deletion), a recovery process is triggered to retrieve data from the backup and restore system data integrity.

[0061] Through the aforementioned technical solutions, the optimized query optimizer can process complex queries faster, significantly reducing query response time. This allows users to obtain analysis results more quickly, enabling them to make more timely decisions. By optimizing the distributed storage architecture and introducing a dynamic load balancing mechanism, our technical solution better supports horizontal scaling. Users can more easily add cluster nodes to cope with ever-increasing data volumes without worrying about performance degradation or increased management complexity. The introduced encryption technology and access control mechanisms provide users with a higher level of data protection, which helps reduce the risk of data leakage and ensures that only authorized users can access sensitive data. At the same time, data backup and recovery functions also provide users with additional data security guarantees.

[0062] In addition, such as Figure 9 As shown, one embodiment of this application also provides a data processing apparatus 10, which includes: Unit 100 is used to retrieve the query statement; The parsing unit 200 is used to parse and process the query statement to obtain statement parsing information; Transformation unit 300 is used to transform the parsed information of the statement according to a preset query rewriting strategy to obtain query transformation information; Construction unit 400 is used to construct a query execution cost model based on the data distribution statistics corresponding to the query transformation information; Evaluation unit 500 is used to perform execution cost evaluation processing based on query transformation information and query execution cost model to obtain multiple execution plan evaluation values; The execution unit 600 is used to take the query execution plan corresponding to the smallest execution plan evaluation value as the target query execution plan, and to perform data query processing based on the target query execution plan.

[0063] The specific implementation of the data processing device 10 is basically the same as the specific embodiment of the data processing method described above, and will not be repeated here.

[0064] In addition, such as Figure 10 As shown, one embodiment of this application also provides an electronic device 700, which includes: a memory 720, a processor 710, and a computer program stored on the memory 720 and executable on the processor 710.

[0065] The processor 710 and memory 720 can be connected via a bus or other means.

[0066] The non-transient software program and instructions required to implement the data processing method of the above embodiments are stored in the memory 720. When executed by the processor 710, the data processing method of each of the above embodiments is executed.

[0067] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0068] Furthermore, one embodiment of this application provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor 710 or a controller, for example, by a processor 710 in the above-described device embodiment, causing the processor 710 to perform the data processing method in the above-described embodiment.

[0069] The above embodiments can be used in combination, and modules with the same name in different embodiments may be the same or different.

[0070] The foregoing has described specific embodiments of this application; other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily have to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0071] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and computer-readable storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0072] The apparatus, device, computer-readable storage medium and method provided in the embodiments of this application are corresponding. Therefore, the apparatus, device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, device and computer storage medium will not be described again here.

[0073] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used when writing program development code. The original code before compilation must also be written in a specific programming language, which is called a Hardware Description Language (HDL). There is not just one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using the aforementioned hardware description languages ​​and programming it into an integrated circuit, the hardware circuit that implements the logic method flow can be easily obtained.

[0074] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0075] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0076] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, in implementing the embodiments of this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0077] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0079] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0080] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0081] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0082] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (FlashRAM). Memory is an example of computer-readable media.

[0083] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0084] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0085] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0086] The embodiments of this application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0087] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0088] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0089] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A data processing method, characterized in that, The data processing method includes: Retrieve the query statement; The query statement is parsed to obtain statement parsing information; The parsed information of the statement is transformed according to a preset query rewriting strategy to obtain query transformation information; A query execution cost model is constructed based on the data distribution statistics corresponding to the query transformation information; Based on the query transformation information and the query execution cost model, an execution cost evaluation process is performed to obtain multiple execution plan evaluation values; The query execution plan corresponding to the smallest execution plan evaluation value is taken as the target query execution plan, and data query processing is performed based on the target query execution plan.

2. The data processing method according to claim 1, characterized in that, The parsing of the query statement to obtain statement parsing information includes: The query statement is parsed to obtain an abstract syntax tree; The abstract syntax tree is subjected to semantic parsing to obtain the parsing information of the statement.

3. The data processing method according to claim 1, characterized in that, The step of transforming the parsed statement information according to a preset query rewriting strategy to obtain query transformation information includes: The parsed information of the statement is converted into a statement relational algebra expression tree; The predicates in the relational algebra expression tree of the statements are normalized to obtain the statement conjunctive normal form; Based on a predefined equivalence transformation rule base, the statement relational algebra expression tree, which converts the predicate into the statement conjunctive normal form, is subjected to equivalence transformation processing to obtain the query transformation information.

4. The data processing method according to claim 1, characterized in that, The step of constructing a query execution cost model based on the data distribution statistics corresponding to the query transformation information includes: Based on the table and column information of the query transformation information, the corresponding data distribution statistics are obtained from the preset system metadata; Based on the data distribution statistics, preset hardware resources, and node status, the query execution cost model is constructed.

5. The data processing method according to claim 1, characterized in that, The execution cost evaluation process, based on the query transformation information and the query execution cost model, yields multiple execution plan evaluation values, including: Multiple logical operations are determined from the query transformation information; Based on the multiple logical operations and the preset operation mapping library, multiple candidate physical operations are determined; Multiple candidate physical operations are enumerated and joined to obtain multiple query execution plans; The query execution cost model is used to evaluate multiple query execution plans to obtain multiple execution plan evaluation values.

6. The data processing method according to claim 1, characterized in that, The data query processing based on the target query execution plan includes: The query execution steps and multiple execution sub-tasks are determined based on the target query execution plan; The query execution steps are used to execute multiple execution subtasks to perform data query processing on a preset database.

7. The data processing method according to claim 6, characterized in that, The database is a distributed database, and the data storage process of the database includes: Receive raw data; Based on the data characteristics of the original data, data fragmentation is performed to obtain multiple fragmented data; Load analysis is performed on multiple nodes corresponding to the distributed database to obtain node load status information. The data shards are distributed and stored based on the node load status information.

8. A data processing apparatus, characterized in that, The data processing device includes: The retrieval unit is used to retrieve the query statement; The parsing unit is used to parse the query statement to obtain statement parsing information; The transformation unit is used to transform the parsed information of the statement according to a preset query rewriting strategy to obtain query transformation information; The construction unit is used to construct a query execution cost model based on the data distribution statistics corresponding to the query transformation information; The evaluation unit is used to perform execution cost evaluation processing based on the query transformation information and the query execution cost model to obtain multiple execution plan evaluation values; The execution unit is used to take the query execution plan corresponding to the smallest execution plan evaluation value as the target query execution plan, and to perform data query processing based on the target query execution plan.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are used to execute the data processing method according to any one of claims 1 to 7.

Images