Data limited query control method and device, equipment and medium

Abstract

The invention relates to a data limited query control method and device, equipment and a medium. The method comprises the steps that a data query statement is intercepted, and a distributed data engine is called to generate a logic plan and a physical plan corresponding to the data query statement to be executed; correspondingly determining whether the data query statement belongs to a limited quick query type or not according to whether tree grammar structure information contained in the logic plan is matched with a preset limited quick query feature or not; when the data query statement does not belong to the limited quick query type, calling and executing the physical plan, and obtaining a query result corresponding to the data query statement in the distributed file system through execution; and when the data query statement belongs to the file path in the physical plan, determining a target data file matched with the data query statement in the distributed file system according to the file path in the physical plan, and executing the data query statement on the target data file to obtain a corresponding query result. The query response speed can be increased, and the resource utilization efficiency can be optimized.

Description

Technical Field

[0001] The present application relates to big data processing technology, and in particular to a data limited query control method and its device, equipment, and medium. Background Art

[0002] In big data processing scenarios such as e-commerce platforms and live streaming platforms, with the explosive growth of data volumes, distributed data engines such as Apache Spark are widely used to process large datasets. These distributed data engines achieve efficient data processing by breaking down data query statements into logical and physical plans before executing query operations. However, this traditional query processing approach presents technical challenges when dealing with certain types of queries.

[0003] Traditionally, when a user submits a data query, the distributed data engine first generates a logical plan. This plan breaks the query down into multiple execution nodes and defines the dependencies between these execution nodes. The engine then generates a physical plan, which specifically describes how to execute the logical plan within the distributed file system. While this process can handle complex queries, it is overly complex and time-consuming for simple queries, especially those that only need to return a small amount of data. This means that even for queries with relatively small amounts of data, these distributed data engines still take a long time to complete the data query process.

[0004] For example, when a query statement contains a LIMIT clause, especially when the query complexity is low, traditional distributed data engines will still execute the query according to the complete logical plan and physical plan. This not only leads to long query response times, sometimes up to 1 hour in some scenarios with huge data volumes, but also consumes unnecessary computing resources.

[0005] Furthermore, traditional technologies are unable to effectively distinguish query complexity when processing these simple queries, requiring even simple queries to undergo a complete query processing flow. This processing approach significantly reduces overall system performance and increases system load when processing high-frequency simple queries, while also impacting the user experience. Consequently, traditional technologies suffer from long query response times and low resource utilization when processing simple queries. Summary of the Invention

[0006] The purpose of this application is to solve the above problems and provide a data limited query control method and its corresponding device, equipment, non-volatile readable storage medium, and computer program product.

[0007] According to one aspect of the present application, a data limited query control method is provided, comprising:

[0008] Intercepting a data query statement, calling a distributed data engine to generate a logical plan corresponding to the data query statement to be executed, and specifically executing a physical plan corresponding to the logical plan in the distributed file system;

[0009] determining whether the data query statement belongs to a limited quick search type according to whether the tree-like grammatical structure information of the data query statement contained in the logical plan matches a preset limited quick search feature;

[0010] When the query does not belong to the limited quick query type, calling the distributed data engine to execute the physical plan, and obtaining the query result corresponding to the data query statement in the distributed file system by executing the physical plan;

[0011] When it belongs to the limited quick search type, the target data file matching the data query statement in the distributed file system is determined according to the file path in the physical plan, and the corresponding query result is obtained by executing the data query statement on the target data file.

[0012] According to another aspect of the present application, a data limited query control device is provided, comprising:

[0013] An interception and analysis module is configured to intercept a data query statement, call a distributed data engine to generate a logical plan corresponding to the data query statement to be executed, and specifically execute a physical plan corresponding to the logical plan in the distributed file system;

[0014] a type determination module configured to determine whether the data query statement belongs to a limited quick search type according to whether the tree grammatical structure information of the data query statement included in the logical plan matches a preset limited quick search feature;

[0015] an engine query module configured to, when the query does not belong to the limited quick query type, call the distributed data engine to execute the physical plan, and obtain a query result corresponding to the data query statement in the distributed file system by executing the physical plan;

[0016] The proxy query module is configured to determine, when belonging to a limited quick search type, a target data file in the distributed file system that matches the data query statement based on the file path in the physical plan, and obtain a corresponding query result by executing the data query statement on the target data file.

[0017] According to another aspect of the present application, a data limited query control device is provided, comprising a central processing unit and a memory, wherein the central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method described in the present application.

[0018] According to another aspect of the present application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the data limit query control method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the method are executed.

[0019] According to another aspect of the present application, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method when executed by a processor.

[0020] Compared with the existing technology, this application effectively solves the problems of long response time and low resource utilization efficiency faced by traditional technologies when processing simple queries by accurately intercepting and analyzing data query statements. After receiving the data query statement, this application first calls the distributed data engine to generate a logical plan and a physical plan, and accurately determines whether the query belongs to the limited quick query type through in-depth analysis of the tree-like grammatical structure information in the logical plan. For simple queries, this application directly locks the target data file in the distributed file system based on the file path in the physical plan, and quickly executes the query to obtain the results, which significantly shortens the query response time and improves the user experience. At the same time, this application avoids excessive occupation of the distributed data engine by accurately identifying and quickly processing simple queries, optimizes resource utilization efficiency, and reduces system operating costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This application is an exemplary network architecture that can be used to deploy and implement product search services;

[0022] Figure 2 This is a flow chart of an embodiment of the data limited query control method of the present application;

[0023] Figure 3 This is a functional block diagram of the data limited query control device of this application;

[0024] Figure 4 This is a structural diagram of a data limited query control device used in this application. DETAILED DESCRIPTION

[0025] like Figure 1As shown, the exemplary network architecture of the present application includes a front-end server 81 and a terminal device 80. The front-end server can be a search service server in the service cluster of the e-commerce platform, which undertakes the core function of providing data query services to the terminal devices. The front-end server 81 runs a computer program product implemented according to the data limited query control method of the present application. The program product can process data query requests carrying data query statements from terminal devices according to the technical solution of the present application, and return relatively accurate query results. It should be pointed out that the terminal device 80 can be either a user terminal or a server with relatively independent functions, such as a middle-end server in an e-commerce platform.

[0026] Furthermore, after the front-end server 81, a distributed file system can be connected and a distributed data engine can be configured to provide a larger network architecture and a distributed data system to support data query services. In modern distributed systems, distributed file systems and distributed data engines are key technologies for processing large-scale data. Distributed file systems, such as the Hadoop Distributed File System (HDFS) or the Google File System (GFS), achieve distributed storage of data by dividing data into multiple shards and storing them on multiple nodes. This sharding technology not only improves data storage efficiency, but also enhances the system's fault tolerance through a data replication mechanism.

[0027] Distributed data engines, such as Apache Spark, further enhance the efficiency and flexibility of data processing. Spark, through its core abstraction of Resilient Distributed Datasets (RDDs), allows for efficient in-memory processing of data, significantly increasing data processing speed. Furthermore, Spark's DAG scheduler divides tasks into multiple stages based on the dependencies between RDDs and efficiently distributes them to various nodes in the cluster for execution.

[0028] In practical applications, such as e-commerce platforms or live streaming platforms, front-end servers are typically responsible for providing data query services to terminal devices. These front-end servers can be deployed in a service cluster and run a computer program product implemented according to the data limited query control method of the present application to provide data query services. This program product can process data query requests carrying data query statements from terminal devices and return accurate query results.

[0029] For example, on an e-commerce platform, the front-end server can quickly respond to user queries, such as searching for products or checking order status, by working in conjunction with the distributed file system and distributed data engine to provide efficient data query services. In this network architecture, the distributed file system is responsible for storing large amounts of product information and user data, while the distributed data engine is responsible for processing and analyzing this data to support fast query responses.

[0030] This architecture allows e-commerce platforms to quickly process user queries and provide real-time product information and order status, enhancing the user experience. Live streaming platforms can also leverage this architecture to quickly process audience interaction requests, such as comments or likes, ensuring smooth and interactive live streaming through the efficient processing capabilities of the distributed data engine.

[0031] By combining the powerful features of a distributed file system and a distributed data engine, the technical solution of this application can provide efficient and reliable data query services for application scenarios such as e-commerce platforms and online live streaming platforms. This architecture not only improves the speed of data processing, but also enhances the stability and fault tolerance of the system through data sharding and replication mechanisms.

[0032] See also Figure 2 According to a data limited query control method provided by the present application, a computer program product can be implemented as a computer program product installed and run on a server to centrally provide data query services. In some embodiments, the method includes the following steps:

[0033] Step S3100: intercepting a data query statement, calling a distributed data engine to generate a logical plan corresponding to the execution of the data query statement, and specifically executing a physical plan corresponding to the logical plan in the distributed file system;

[0034] In a distributed data processing system, a data query script can be submitted to the front-end server based on various uses and purposes. The data query script can contain a single data query statement, or directly encapsulate a single data query statement through a data query request and send it to the front-end server. After the front-end server receives the data query statements submitted by the user, it first needs to intercept these query statements. The purpose of interception is to perform a preliminary analysis of the query statements to determine whether they are suitable for an optimized query processing process. The interception operation can be implemented by deploying a special interception module on the front-end server, which can be a software component responsible for capturing and processing all query requests entering the system. It can be seen that the front-end server actually plays the role of a proxy server that processes data query statements between the terminal device and the distributed data engine.

[0035] After the interception module captures a data query statement, it does not process it directly, but instead submits it to a distributed data engine, such as Apache Spark. After receiving the query statement, the distributed data engine initiates a parsing process to generate a logical plan. The logical plan is a high-level abstract representation of the query statement. It decomposes the query statement into multiple execution nodes and defines the dependencies between these nodes, expressed as a tree-like grammatical structure. For example, for the query statement SELECT * FROM table WHERE condition LIMIT 10, the logical plan may include a Project node (for projection operations), a Filter node (for filtering operations), and a Limit node (for limiting the number of returned results).

[0036] After generating the logical plan, the distributed data engine further generates a physical plan. The physical plan is the specific implementation of the logical plan, describing how the query will actually be executed in the distributed file system. The physical plan includes information such as the data read path, the specific implementation of the filter conditions, and data partition information. For example, for the query above, the physical plan might specify reading data from a specific file path in HDFS, applying the filter conditions, and returning the first 10 data records that meet the conditions.

[0037] Step S3200: determining whether the data query statement belongs to a limited quick search type based on whether the tree-like grammatical structure information of the data query statement included in the logical plan matches a preset limited quick search feature;

[0038] After capturing a data query statement and submitting it to the distributed data engine for logical plan generation, the interception module retrieves the logical plan and analyzes the tree-structured grammatical information within it. This tree-structured grammatical information is the core of the logical plan. Each execution node in the tree represents an operation in the data query statement, such as projection, filtering, or restriction.

[0039] To determine whether a data query statement is a limited-query type, the interception module checks whether the tree-like grammatical structure of the logical plan matches a predefined limited-query signature. A limited-query signature can be one or more rules defined based on the structure and content of the data query statement, used to identify simple queries with low query complexity and returning only a small amount of data. For example, a typical limited-query signature is a data query statement containing a LIMIT clause and exhibiting low query complexity, without complex nested subqueries, aggregation operations, or cross-table projections.

[0040] In one embodiment, the interception module may first check whether the top-level execution node of the logical plan matches a first limited quick-check feature. The first limited quick-check feature is typically used to limit the number of data queries required by a data query statement, for example, whether the data query statement contains a LIMIT clause. If the top-level execution node does not match the first limited quick-check feature, the interception module will determine that the data query statement does not belong to the limited quick-check type.

[0041] If the top-level execution node matches the first limited quick-check feature, the interception module can further check whether there is an execution node matching the second limited quick-check feature in the logical plan. The second limited quick-check feature is used to indicate whether the query complexity of the data query statement exceeds the expected conditions. The expected conditions may include whether the query scope exceeds a single data table, whether the number of filter conditions exceeds a preset value, whether there are nested subqueries, aggregation operations, window functions, or projection operations across data tables, etc., and any one or more of the above conditions can be preset according to the requirements for the complexity of the data query statement. If there is an execution node matching the second limited quick-check feature in the logical plan, the interception module will determine that the data query statement does not belong to the limited quick-check type.

[0042] The interception module determines that a data query statement belongs to the limited quick-check type only if there are no execution nodes matching the second limited quick-check feature in the logical plan. This indicates that the data query statement is relatively simple and only needs to return a small amount of data, making it suitable for optimized query processing.

[0043] Step S3300: If the query does not belong to the limited quick search type, the distributed data engine is called to execute the physical plan, and query results corresponding to the data query statement are obtained in the distributed file system by executing the physical plan.

[0044] If the interception module determines that a data query statement does not fall into the limited quick-query category, it indicates that the query statement is complex and unsuitable for optimized query processing. In this case, the distributed data engine must be invoked to execute the physical plan to ensure that the query is correctly executed and returns results. Distributed data engines, such as Apache Spark, are capable of handling complex queries and can efficiently execute query operations in a distributed file system according to their physical plan.

[0045] Specifically, the distributed data engine executes queries based on information in the physical plan, such as the data's file path (read path), the specific implementation of filter conditions, and data partition information. The physical plan is a detailed implementation of the logical plan, describing how the query is actually executed in the distributed file system. For example, for a complex data query, the physical plan might specify reading data from multiple file paths in HDFS, applying multiple filter conditions, and performing data aggregation operations.

[0046] When executing a physical plan, the distributed data engine launches multiple tasks, which are distributed and executed in parallel across the nodes in the cluster. Each task processes a portion of the data and ultimately aggregates and returns the results to the front-end server. For example, for a query involving a join operation on multiple tables, the physical plan will detail how the data is read and processed on each node, as well as how the results are merged.

[0047] To achieve this, the distributed data engine utilizes its built-in scheduler and resource manager to optimize task execution. The scheduler arranges the execution order of tasks based on task dependencies and resource availability, while the resource manager allocates computing resources, such as CPU and memory, to ensure efficient task execution. For example, Spark's DAG scheduler divides tasks into multiple stages based on the dependencies between RDDs and efficiently distributes them to various nodes in the cluster for execution.

[0048] In certain embodiments, the distributed data engine can employ data caching techniques to improve query performance. For example, Spark can cache frequently accessed data in memory, allowing subsequent queries to quickly retrieve the data. Furthermore, the distributed data engine can utilize data partitioning techniques to optimize data reading and processing. For example, data in HDFS can be partitioned according to specific rules, and the distributed data engine can efficiently read data based on this partitioning information.

[0049] Step S3400: When it belongs to the limited quick search type, determine the target data file in the distributed file system that matches the data query statement according to the file path in the physical plan, and obtain the corresponding query result by executing the data query statement on the target data file.

[0050] When the interception module determines that a data query statement is a limited quick-query type, it means that the query statement is relatively simple and only requires a small amount of data to be returned. In this case, to improve query response speed and optimize resource utilization, instead of directly calling the distributed data engine to execute the physical plan, a more efficient query processing method is adopted.

[0051] Specifically, the front-end server determines the target data file in the distributed file system that matches the data query statement based on the file path provided in the physical plan. The physical plan describes in detail the storage location of the data in the distributed file system through file path information, including specific file paths, partition paths, etc. For example, for data stored in the Hadoop Distributed File System (HDFS), the file path may be similar to " / user / hive / warehouse / table / dt=20250430 / ", where " / user / hive / warehouse / table / " is the root path of the table, and "dt=20250430" represents data corresponding to the date April 30, 2025, and the target data file corresponding to that day can be specified.

[0052] After identifying the target data files, the next step is to execute data query statements against these target data files to obtain query results. Since the query statements have been identified as limited and quick-search types, some optimization measures can be used to quickly obtain results. For example, data in the target data files can be directly read without going through the full execution process of the distributed data engine.

[0053] In specific implementation, the client tools or APIs provided by the distributed file system can be used to read files. Taking HDFS as an example, the Java API or command-line tools of HDFS can be used to access and read files. When reading data, the data records that meet the conditions are filtered and extracted according to the filter conditions and projection columns in the data query statement. For example, if the query statement is "SELECT name,age FROM people WHERE age>20LIMIT 10", then when reading the target data file, only the data of the "name" and "age" columns are extracted, and only those records with "age" greater than 20 are retained. When the number of data records read reaches the number specified in the query statement, such as 10 records, the reading operation is stopped immediately, thereby further improving the query efficiency.

[0054] This approach allows for quick query results without having to wait for the distributed data engine to complete the execution of the entire physical plan. This significantly reduces query response time and reduces the use of distributed computing resources, improving overall system performance. For example, on an e-commerce platform, a user might need to quickly view the 10 most recent order records. This optimized query processing method can return results in as little as one second, rather than waiting for several minutes or even longer. Furthermore, this approach can also be applied to other scenarios requiring fast responses to simple queries, such as displaying the number of likes or comment content from viewers on live streaming platforms in real time.

[0055] In practical applications, to further improve query efficiency, some preprocessing operations can be performed on the target data files. For example, data can be pre-sorted or indexed before being written to the distributed file system, allowing for faster location of matching records during queries. Furthermore, hot data can be cached in memory based on data access patterns and query frequency, further reducing read latency.

[0056] As can be seen from the above examples, compared to the existing technology, this application cleverly solves the problems of long response time and low resource utilization efficiency faced by traditional technologies when processing simple queries by accurately intercepting and analyzing data query statements. Specifically, after receiving the data query statement, this application does not directly put it into the distributed data engine for execution, but first calls the distributed data engine to generate a logical plan and a physical plan. This process seems to add steps, but in fact it lays the foundation for subsequent efficient processing.

[0057] By conducting an in-depth analysis of the tree-like grammatical structure information in the logical plan, rather than directly analyzing the data query statements, this application can effectively avoid various potential anomalies in the original expression of the data query statements, and use the distributed data engine's deep understanding of the data query statements to accurately determine whether the data query statements belong to the limited quick-check type. For those simple queries that only need to return a small amount of data, this application no longer lets them go through the traditional and complex query execution process, but directly locks the target data files in the distributed file system based on the file path in the physical plan. Subsequently, the data query statements are directly executed on these target data files to quickly obtain the query results. This innovative processing method greatly shortens the query response time, allowing users to get the required data instantly, significantly improving the user experience.

[0058] At the same time, this application not only improves query response speed, but also greatly optimizes resource utilization efficiency. In traditional technologies, even simple queries will occupy a large amount of computing resources, resulting in resource waste. However, this application avoids excessive occupation of the distributed data engine by these queries by accurately identifying and quickly processing simple queries, allowing limited computing resources to be more efficiently allocated to other complex query tasks, thereby improving the resource utilization efficiency of the entire system and reducing the operating costs of the system.

[0059] Based on any embodiment of the method of the present application, calling a distributed data engine to generate a logical plan corresponding to the data query statement to be executed, and specifically executing a physical plan corresponding to the logical plan in the distributed file system, includes:

[0060] Step S3110: submitting the data query statement to the distributed data engine for lazy execution to trigger generation of the logical plan and the physical plan, but not executing the physical plan;

[0061] When the interception module of the front-end server captures the data query statement submitted by the user, it needs to be submitted to the distributed data engine for further analysis and processing of the query statement. Distributed data engines, such as Apache Spark, have powerful data processing capabilities and can parse and optimize query statements. In this step, the interception module submits the data query statement to the distributed data engine, and implements lazy execution according to the execution mechanism of the distributed data engine. Lazy execution means that after receiving the query statement, the distributed data engine will not execute the query immediately, but will first generate a logical plan and a physical plan. This execution method allows the distributed data engine to deeply parse and optimize the data query statement, providing a basis for subsequent query processing.

[0062] A logical plan is a high-level, abstract representation of a query statement. It breaks the query statement down into multiple execution nodes and defines the dependencies between these nodes. For example, for the query statement SELECT * FROM table WHERE condition LIMIT 10 , the logical plan might include a Project node (for projection), a Filter node (for filtering), and a Limit node (for limiting the number of returned results). These execution nodes together form a tree-like grammatical structure that details the semantics and execution logic of the query statement.

[0063] The physical plan is the specific implementation of the logical plan, describing how the query is actually executed in the distributed file system. It includes information such as the data read path, the specific implementation of the filter conditions, and data partitioning information. For example, for the query above, the physical plan might specify reading data from a specific file path in HDFS, applying the filter conditions, and returning the first 10 records that meet the conditions.

[0064] In practical applications, submitting data query statements to a distributed data engine for lazy execution can be achieved by calling the API provided by the distributed data engine. Taking Apache Spark as an example, you can use the spark.sql() method to submit a data query statement, for example: val df = spark.sql("SELECT * FROM table WHERE condition LIMIT10"). The spark.sql() method submits the data query statement, generates a corresponding logical plan based on it, and then generates a physical plan based on the logical plan, but does not immediately execute the query. You can obtain the logical plan by calling df.queryExecution.analyzed, and the physical plan by calling df.queryExecution.executedPlan.

[0065] Step S3120: Calling a logical plan acquisition interface to acquire the logical plan. The logical plan includes tree-like grammatical structure information. The tree-like grammatical structure information decomposes the data query statement into multiple execution nodes and defines dependencies between the execution nodes. At least one execution node includes the name of a target data table to be operated on by the data query statement.

[0066] When the interception module submits a data query statement to the distributed data engine for lazy execution, the distributed data engine generates a logical plan. This logical plan is a high-level, abstract representation of the query statement. It breaks the query statement into multiple execution nodes and defines the dependencies between these nodes. These execution nodes together form a tree-like grammatical structure that details the query statement's semantics and execution logic.

[0067] You can retrieve the logical plan by calling the logical plan retrieval interface. This interface is typically an API provided by the distributed data engine, allowing users or the system to retrieve the generated logical plan. The logical plan contains rich information, such as the projection operations (Project), filter operations (Filter), and limit operations (Limit) in the query statement. These operations are represented as execution nodes in the logical plan.

[0068] For example, for the query "SELECT name,age FROM people WHERE age>20 LIMIT 10", the logical plan may include the following nodes:

[0069] Project: represents the projection operation, which is used to select the "name" and "age" columns.

[0070] Filter: Indicates a filtering operation, used to apply the condition "age>20".

[0071] Limit: indicates a limit operation, which is used to limit the number of returned results to 10.

[0072] The dependencies between these nodes define the order in which queries are executed. For example, a Filter node may depend on a Project node, indicating that the projection operation should be performed first, followed by the filter operation.

[0073] After obtaining the logical plan, you can further analyze the tree-like grammatical structure to determine whether the query statement meets the requirements for limited-query querying. Limited-query querying refers to queries that are relatively simple and only return a small amount of data. For example, the query statement contains a LIMIT clause and does not involve complex nested subqueries, aggregation operations, or cross-table projections.

[0074] In practice, obtaining the logical plan can be achieved by calling the API provided by the distributed data engine. For the example above, you can use the df.queryExecution.analyzed method to obtain the logical plan and assign it to the corresponding variable, for example: val logicalPlan = df.queryExecution.analyzed . Here, the logicalPlan variable will contain the generated logical plan, which can be further analyzed to determine whether the query statement is a limited quick query type.

[0075] Step S3130: Call a physical plan acquisition interface to acquire the physical plan, where the physical plan includes a file path of a target data file in the distributed file system determined according to the table name of the target data table.

[0076] After the interception module submits the data query statement to the distributed data engine for lazy execution, the distributed data engine generates a physical plan. The physical plan is the specific execution plan of the logical plan, detailing how the query is actually executed in the distributed file system. The physical plan includes the data reading path, the specific implementation of the filter conditions, data partitioning information, and more. For example, for the query statement "SELECT name, age FROM people WHERE age>20LIMIT10", the physical plan can specify reading data from a specific file path in HDFS, applying the filter conditions, and returning the first 10 data records that meet the conditions.

[0077] To obtain the physical plan, you can call the physical plan acquisition interface provided by the distributed data engine. This interface is typically an API provided by the distributed data engine, allowing users or the system to obtain the generated physical plan. Continuing with the previous example, you can obtain the physical plan by calling df.queryExecution.executedPlan. The physical plan contains rich information, such as the data file path, partition path, and the specific implementation of the filter conditions.

[0078] The file path information in the physical plan is critical, as it specifies where the data is stored in the distributed file system. For example, a physical plan may contain the following information:

[0079] · File path: / user / hive / warehouse / people / dt=20250430 /

[0080] · Partition path: dt=20250430

[0081] · Filter condition: age>20

[0082] · Number of results returned: LIMIT 10

[0083] This information can be directly located in the target data file in the distributed file system, and data can be read and processed according to the query statement. For example, if the data query statement is "SELECT name, age FROM people WHERE age>20 LIMIT 10", the physical plan will specify to read data from the path / user / hive / warehouse / people / dt=20250430 / , apply the filter condition age>20, and return the first 10 data records that meet the condition.

[0084] By calling the physical plan acquisition API to obtain the physical plan, you can obtain all the detailed information required to execute the query. This information includes not only the file path and partition path of the data, but also the specific implementation of the filter conditions and the limit on the number of returned results. This information allows you to directly locate the target data file, read and process the data according to the query statement, thereby achieving fast query response and optimizing resource utilization.

[0085] In this embodiment, the front-end server interception module submits the data query statement to the distributed data engine for lazy execution, generates a logical plan and a physical plan, but does not execute a specific data query. This process significantly improves the efficiency and accuracy of query processing. First, with the help of the powerful parsing and optimization capabilities of the distributed data engine, the generated logical plan and physical plan provide a detailed and structured information basis for subsequent query analysis, so that the front-end server can accurately identify whether the query statement belongs to the limited quick query type, thereby avoiding misjudgment of complex query statements and unnecessary resource consumption. Secondly, the tree-like grammatical structure information of the logical plan clearly decomposes the execution logic of the data query statement and can efficiently analyze the complexity of the query, while the specific execution information such as the file path in the physical plan provides clear guidance for directly reading the target data file, further optimizing the query execution path. This processing method not only greatly shortens the response time of simple queries and improves the user experience, but also releases the computing resources of the distributed data engine by accurately identifying and optimizing simple queries, enabling it to process complex query tasks more efficiently, and significantly improving the resource utilization efficiency and query processing performance of the system as a whole.

[0086] Based on any embodiment of the method of the present application, determining whether the data query statement belongs to the limited quick search type according to whether the tree grammatical structure information of the data query statement included in the logical plan matches the preset limited quick search feature includes:

[0087] Step S3210: Determine whether the top-level execution node in the tree-like grammatical structure information of the logical plan matches a first limited quick-check feature. If the first limited quick-check feature is not matched, determine that the data query statement does not belong to the limited quick-check type. The first limited quick-check feature is used to limit the number of data queries required by the data query statement.

[0088] As previously mentioned, the tree-like grammatical structure of the logical plan is a high-level, abstract representation of the query statement. It breaks the query statement into multiple execution nodes and defines the dependencies between these nodes. Each execution node represents an operation in the query statement, such as projection, filtering, or restriction. In the tree structure, the top-level execution node is typically the final operation of the entire query, which determines the query's final output.

[0089] Based on this, we first check whether the top-level execution node of the logical plan contains a LIMIT clause or other similar restriction. If the top-level execution node does not match the first limited quick query characteristic, that is, the query statement does not explicitly limit the number of returned results, then we can quickly determine that the data query statement does not belong to the limited quick query type. This is because the core characteristics of the limited quick query type are low query complexity and only a small amount of data needs to be returned, and queries without a LIMIT clause generally do not meet this characteristic.

[0090] For example, for the query statement SELECT * FROM table WHERE condition on LIM IT 10, the top-level execution node in the logical plan is a Limit node, indicating that the query statement limits the number of returned results to 10. In this case, the top-level execution node in the logical plan matches the first limited quick-check feature. Conversely, for the query statement SELECT * FROM table WHERE condition, since there is no LIM IT clause, the top-level execution node may be a Project or Filter node, and therefore does not match the first limited quick-check feature.

[0091] Step S3220: When the first limited quick search feature is matched, determining whether there is an execution node matching the second limited quick search feature in the tree-like grammatical structure information; if the execution node exists, determining that the data query statement does not belong to the limited quick search type, and the second limited quick search feature is used to indicate that the query complexity of the data query statement exceeds the expected condition;

[0092] When the top-level execution node of the logical plan matches the first limited quick-check feature, that is, the query statement contains a LIMIT clause or other similar restriction operations, the next step is to determine whether there is an execution node that matches the second limited quick-check feature in the tree-like grammatical structure information of the logical plan. In this application, the second limited quick-check feature is used to indicate whether the query complexity of the data query statement exceeds the expected conditions. The expected conditions can be set in advance as needed, including but not limited to whether the query scope exceeds a single data table, whether the number of filter conditions exceeds a preset value, whether there are nested subqueries, aggregation operations, window functions or projection operations across data tables, etc., any one or more of them. The following provides multiple embodiments of the expected conditions to provide a deep understanding of the implementation of the second limited quick-check feature.

[0093] In one embodiment, the expected condition indicated by the second limited quick search feature is set to a query scope exceeding a single data table. This means that if the execution node in the logical plan indicates that the query statement involves a join operation on multiple data tables, such as joining multiple tables through a JOIN clause, then the query statement is highly complex and does not belong to the limited quick search type. In this case, the query not only needs to process the data of a single table, but also needs to match and integrate data between multiple tables, which significantly increases the complexity and execution time of the query. Therefore, when there is a join operation involving multiple data tables in the logical plan, the query statement is considered to exceed the expected conditions for limited quick search.

[0094] In one embodiment, the expected conditions indicated by the second limited quick search feature are set to include more than a preset number of filter conditions. Specifically, if the execution node in the logical plan contains multiple complex filter conditions, such as conditions in multiple WHERE clauses, which may involve comparisons of multiple fields, function calls, etc., then the complexity of the query will increase significantly. The preset number can be adjusted according to the actual application scenario and system performance requirements. When the number of filter conditions exceeds this preset value, the query statement is considered to exceed the expected conditions for limited quick search, because too many filter conditions will increase the computational burden of the query and extend the query response time.

[0095] In one embodiment, the expected conditions indicated by the second limited quick query feature are set to include nested subqueries. A nested subquery refers to a query statement that is nested in another query statement, such as SELECT*FROM(SELECT*FROMtable WHERE condition1)WHERE condition2. Nested subqueries increase the complexity of the query because the inner query needs to be executed first and then the result is used as the input of the outer query. This nested structure not only increases the difficulty of query parsing and execution, but may also cause the query response time to increase significantly. Therefore, when the execution node in the logical plan is checked to confirm the existence of a nested subquery, the data query statement is considered to exceed the expected conditions of the limited quick query.

[0096] In one embodiment, the expected conditions indicated by the second limited quick query feature are set to include aggregation operations. Aggregation operations refer to summarizing or statistical calculations on data, such as SUM, COUNT, AVG, etc. These operations usually require processing large amounts of data and may involve complex grouping and calculation logic. If the execution node in the logical plan contains aggregation operations, the complexity of the query will increase significantly because the aggregation operations require scanning and calculating large amounts of data. Therefore, when there are aggregation operations in the logical plan, the query statement is considered to exceed the expected conditions for limited quick query.

[0097] In one embodiment, the expected conditions indicated by the second limited quick query feature are set to include window functions. Window functions (such as ROW_NUMBER, RANK, etc.) are used to perform complex row-level calculations in the query result set, such as calculating row numbers, rankings, etc. The use of these functions will increase the complexity of the query because they require additional processing and sorting of the result set. If the execution node in the logical plan contains a window function, the complexity of the query will increase significantly because the window function requires scanning and calculating a large amount of data. Therefore, when a window function exists in the execution node of the logical plan, the data query statement is considered to exceed the expected conditions for limited quick query.

[0098] In one embodiment, the expected conditions indicated by the second limited quick query feature are set to include cross-data table projection operations. Cross-data table projection operations refer to selecting data from multiple data tables and merging them, such as SELECT table1.column1, table2.column2 FROM table1, table2. This operation will increase the complexity of the query because data needs to be read from multiple tables and merged. If the execution node in the logical plan contains cross-data table projection operations, the complexity of the query will increase significantly because data from multiple tables needs to be processed and integrated. Therefore, when there are cross-data table projection operations in the logical plan, the data query statement is considered to exceed the expected conditions of the limited quick query.

[0099] In practice, the expected conditions included in the second limited quick-check feature can be selected according to the various embodiments listed above to include any number of expected conditions by selecting multiple embodiments as needed. In a typical embodiment, the second limited quick-check feature can include all the expected conditions corresponding to the implementation of all the above embodiments.

[0100] If the logical plan contains an execution node matching the second limited quick-check feature, indicating that the query complexity exceeds the expected criteria, the interception module will determine that the data query statement does not qualify as a limited quick-check. This indicates that the query is too complex to be processed using the optimized query processing flow. Instead, the distributed data engine should be invoked to execute the physical plan to ensure correct query execution and return results.

[0101] Step S3230: When there is no execution node matching the second limited quick search feature, it is determined that the data query statement belongs to the limited quick search type.

[0102] When there is no execution node matching the second limited quick-check feature in the tree-like grammatical structure information of the logical plan, that is, the complexity of the query statement does not exceed the expected conditions and also meets the requirements of the first limited quick-check feature, the interception module determines that the data query statement belongs to the limited quick-check type. This means that the structure of the query statement is relatively simple, only a small amount of data needs to be returned, and it does not involve complex query operations such as multi-table joins, a large number of filter conditions, nested subqueries, aggregation operations, window functions, or cross-data table projection operations. Therefore, this query statement is suitable for an optimized query processing process, directly reading the target data file from the distributed file system and quickly returning the query results, thereby significantly improving the query response speed and optimizing resource utilization efficiency.

[0103] The above embodiment accurately analyzes the logical plan of a data query statement and effectively distinguishes the complexity of the data query statement based on preset limited quick-check features, thereby achieving accurate determination of the query type. First, by checking whether the top-level execution node of the logical plan matches the first limited quick-check feature, that is, whether it contains an operation that limits the number of returned results, queries that may belong to the limited quick-check type are quickly screened. Subsequently, the logical plan is further checked to see whether there are execution nodes that match the second limited quick-check feature. These features cover multiple dimensions such as query scope, number of filter conditions, nested subqueries, aggregation operations, window functions, and cross-table projection operations, comprehensively assessing the complexity of the query statement. If the query statement does not exceed these expected conditions, it is determined to belong to the limited quick-check type. An optimized query processing flow can then be used to directly read the target data file from the distributed file system and quickly return query results. This process not only significantly improves query response speed and meets the demand for fast data retrieval, but also optimizes resource utilization efficiency, avoids excessive occupation of the distributed data engine, and can more efficiently process simple queries while ensuring that complex queries can be correctly executed. Through this hierarchical determination mechanism, the present application achieves efficient, flexible and accurate query type identification, effectively improving the overall performance and user experience of the entire data query system.

[0104] Based on any embodiment of the method of the present application, determining a target data file in the distributed file system that matches the data query statement according to the file path in the physical plan, and obtaining a corresponding query result by executing the data query statement on the target data file includes:

[0105] Step S3410: extracting the file path of the target data file corresponding to the data query statement from the physical plan;

[0106] As mentioned above, the physical plan is a specific execution plan of the logical plan, which describes in detail how to actually execute the query in the distributed file system. The physical plan contains a wealth of information, such as the data reading path, the specific implementation of the filter conditions, the data partition information, etc. In Apache Spark, the physical plan can be obtained by calling df.queryExecution.executedPlan. The file path information in the physical plan is critical, which specifies the storage location of the data in the distributed file system. For example, for data stored in the Hadoop Distributed File System (HDFS), the file path may be similar to / user / hive / warehouse / table / dt=20250430 / , where / user / hive / warehouse / table / is the root path of the table and dt=20250430 is the partition path, indicating that the partition corresponds to data with a date of April 30, 2025.

[0107] In practical applications, file path extraction can be achieved by analyzing the structure of the physical plan. A physical plan is typically a tree structure, with each node representing a query operation. To extract file paths, we typically focus on nodes related to data reading, such as the FileScan node. These nodes contain information such as the file path and partition path. For example, in Spark, a FileScan node might contain the following information:

[0108] · File path: / user / hive / warehouse / table / dt=20250430 /

[0109] · Partition path: dt=20250430

[0110] · Filter condition: age>20

[0111] · Number of results returned: LIMIT 10

[0112] By analyzing this information, you can accurately locate the target data file. For example, if the query statement is SELECT name,age FROM people WHERE age>20 LIMIT 10, the physical plan will specify to read data from the / user / hive / warehouse / people / dt=20250430 / path, apply the filter condition age>20, and return the first 10 data records that meet the condition.

[0113] Step S3420: Connect to the distributed file system through the distributed file system client according to the file path, and locate the target data file;

[0114] After extracting the target data file's file path from the physical plan, a distributed file system client can be used to accurately locate the target data file based on the file path. A distributed file system client is a software component that interacts with a distributed file system, providing the necessary interfaces and tools to read and manipulate data stored in the distributed file system. For example, in the Hadoop Distributed File System (HDFS), client functionality can be implemented using the HDFS Java API or command-line tools. These client tools allow the system to access and read data files based on their file paths.

[0115] To this end, the file path information in the physical plan is parsed, and then the distributed file system client interface is used to establish a connection. The existence of the target path is further checked. After confirming that the file path exists, the status information of all files under the path is obtained to locate the target data file.

[0116] Step S3430: Read the data in the target data file, filter and extract data records that meet the conditions according to the data query statement;

[0117] After locating the target data files based on the file paths in the physical plan, the next step is to read the data from these files. This read operation can also be performed using the distributed file system client. During the data reading process, when a data query statement includes filter conditions and projection columns, data can be filtered and extracted based on the filter conditions and projection columns in the query statement. Filter conditions determine which data records meet the query requirements, while projection columns specify the data fields to be returned. For example, the query statement "SELECT name, age FROM people WHERE age>20LIMIT10" reads the data from the target data file, then applies the filter condition "age>20" to filter the data records. Based on the "LIMIT 10" condition, only 10 data records from the "name" and "age" columns are extracted.

[0118] To further optimize the reading process, various techniques can be employed. For example, data partitioning can be leveraged to reduce the amount of data required for reading. If data is stored by date partition, such as " / user / hive / warehouse / people / dt=20250430 / ," you can directly locate the specific partition path, avoiding reading the entire dataset. Furthermore, data indexing can be leveraged to accelerate data locating and reading.

[0119] When reading data, you can read the data file row by row or block by block and apply filters. When the number of read data records reaches the limit specified in the query statement, such as "LIMIT 10", the reading operation stops immediately, further improving query efficiency. This on-demand reading method ensures that the system does not process redundant data, saving computing resources and time.

[0120] Step S3440: Return the filtered and extracted data records as query results.

[0121] After locating the target data file and reading the data, the data records that meet the conditions in the data query statement have been filtered and extracted. These data records are the results required by the data query statement, and the number of data records meets the query limit, such as LIMIT 10. To return these results to the requesting party, the filtered and extracted data records can be packaged and transmitted.

[0122] Specifically, these data records can be organized into a data structure, such as a table or a DataFrame, for easier transmission and further processing. This data structure contains all the necessary information for the query results, such as column names, data types, and data values. For example, the query statement SELECT name, age FROM people WHERE age > 20 LIMIT 10 will organize the 10 filtered data records into a table containing the name and age columns.

[0123] In practice, query results can be returned in a variety of ways. For example, if the query request is submitted through a web interface, the query result can be encapsulated as a JSON object and returned. If the query request is submitted through an internal API, the query result can be encapsulated as a data frame and returned to the requester. Regardless of the method, the returned query result should contain all the necessary information so that the requester can correctly parse and use the data.

[0124] The above embodiment extracts the file path of the target data file from the physical plan, uses the distributed file system client to connect to the distributed file system and locate the target data file, and then reads the data in the target data file and filters and extracts the data records that meet the conditions according to the data query statement, and finally returns these data records as query results, thereby realizing an efficient and resource-optimized data query processing process. This process significantly improves the query response speed because by directly reading the target data file and applying the filter conditions and projection columns, it avoids the full scanning and processing of the entire data set, reducing unnecessary data reading and calculation. At the same time, by reading data on demand and stopping the reading operation in time, computing resources and time are further saved. In practice, the response time of limited query events triggered in the distributed data system can be reduced to less than seconds. In addition, the filtered and extracted data records are encapsulated into a data structure that is easy to transmit and process, ensuring the accuracy and completeness of the query results, allowing the requester to quickly obtain the required data, thereby improving the performance of the entire system and user experience.

[0125] Based on any embodiment of the method of the present application, after obtaining a corresponding query result by executing the data query statement on the target data file, the method includes:

[0126] Step S4100: Associating the data query statement, the query result, and the current timestamp corresponding to the query result, and storing them in a cache system;

[0127] After executing a data query statement and obtaining query results from the target data file, this information can be stored in a cache system to quickly respond to subsequent requests for the same query. When storing the query results, the cache system also stores the query statement itself and the timestamp of the result. The timestamp records the time when the query result was generated, allowing for subsequent determination of the freshness of the query result.

[0128] Specifically, corresponding to the current query result, a cache entry is created in the cache system, which contains the following information: data query statement: This is the original query statement submitted by the user, which is used to match the same query request in the future; query result: This is the data record obtained after executing the query statement, which is the information requested by the user; timestamp: This is the system time when the query result is generated, which is used to determine whether the query result is expired in the future.

[0129] In practice, caching systems can be implemented in a variety of ways. For example, you can use an in-memory cache, such as Redis or Memcached. These cache systems offer fast read and write performance and are suitable for storing frequently accessed data. Another implementation is to use a local disk cache. While this offers relatively slow read and write speeds, it can store more data. Furthermore, a combination of in-memory and disk caches can be used to balance performance and storage capacity requirements.

[0130] In one embodiment, a data query statement can be converted into a unique hash value and used as the key in a key-value pair in the cache system. The query result and timestamp corresponding to the data query statement are concatenated and stored as the value in the key-value pair. In this way, subsequent data query statements can quickly retrieve the corresponding query results using their hash value.

[0131] Step S4200: For a subsequently intercepted data query statement, first perform a query in the cache system to determine query results and their timestamps associated with the same data query statement;

[0132] When the system receives subsequent data query statements, the interception module can first submit these data query statements to the cache system for query. The cache system stores the query results generated by previous query executions and their associated timestamp information. Each cache entry contains the data query statement, query result, and timestamp. The timestamp records the time when the query result was generated and is used to determine the freshness of the query result later.

[0133] Upon querying, if the cache system contains the data query statement being queried, the corresponding query statement and its timestamp can be retrieved. For example, using the previous example, the interception module can convert the subsequently intercepted data query statement into a hash value and search the cache system for the corresponding cache entry. If a matching hash value is found, the cache system returns the relevant query result and its timestamp.

[0134] Step S4300: Determine whether the timestamp retrieved from the cache system exceeds a preset period. If it does not exceed the preset period, respond to the subsequently intercepted data query statement with a corresponding query result.

[0135] When the interception module retrieves the query results and their timestamps associated with the subsequently intercepted data query statements from the cache system, it continues to determine whether the timestamp exceeds the preset period. The preset period is a time threshold used to determine whether the cached query results are still valid. If the difference between the generation time of the timestamp record and the current time is less than or equal to the preset period, it means that the cached query results are fresh and can be used to directly answer subsequent query requests. On the contrary, if the difference between the generation time of the timestamp record and the current time is greater than the preset period, it means that the cached query results may have expired and no longer have reference value, and the query operation needs to be re-executed to obtain the latest data. For these expired cache records, they can be directly deleted from the cache system to achieve cache cleanup.

[0136] In practice, the preset period can be adjusted based on the specific application scenario and data update frequency. For example, on an e-commerce platform, if product information is frequently updated, the preset period can be set to a shorter period, such as a few minutes or hours; whereas in scenarios where data updates are less frequent, the preset period can be set to a longer period, such as a few days or weeks. This approach ensures the freshness of query results while maximizing cache utilization to improve query efficiency.

[0137] If the timestamp is within the preset period, the interception module can directly use the query results in the cache to answer subsequent intercepted data queries. This step avoids repetitive execution of the same query operation, significantly reducing the system's computational burden and response time. For example, if a user queries the same data multiple times in a short period of time, the results can be returned directly from the cache, without having to access the distributed file system or call the distributed data engine each time.

[0138] This embodiment provides an efficient response mechanism for subsequent query requests by associating data query statements, query results, and the timestamp of the generation time and storing them in the cache system. When the same data query statement is received subsequently, the corresponding query result and its timestamp can be quickly retrieved from the cache, and the freshness of the query result is determined by judging whether the timestamp exceeds the preset period. If it does not exceed the preset period, the query result in the cache is directly used to respond, avoiding repeated execution of the query operation, and significantly reducing the computing burden and response time of the front-end server and distributed data engine. This mechanism is particularly suitable for processing high-frequency simple query requests, and can effectively improve the overall performance and user experience of the entire distributed data system. At the same time, by reasonably setting the preset period, it is possible to maximize the use of cache resources while ensuring the freshness of the query results, further optimizing resource utilization efficiency.

[0139] Based on any embodiment of the method of the present application, after obtaining a corresponding query result by executing the data query statement on the target data file, the method includes:

[0140] Step S5100: Push the query results to the computer device that submitted the data query statement. The computer device encapsulates the query results in a preset format and encapsulates each data record in the query results into an interface component. The data record includes order information from the e-commerce platform.

[0141] After executing the data query statement and obtaining the query results from the target data file, these query results can be pushed to the computer device that submitted the data query statement. This process ensures that the query results can be returned to the requester in a timely manner for further processing and display. In this step, the query results are pushed to the computer device of the requester, which can be a server of a corresponding platform in various specific application scenarios. For example, it can be a server of an independent station deployed by an e-commerce platform. Then, this computer device can package the query results according to a preset format. The packaging process involves organizing the various data records in the query results into a format suitable for display, for example, packaging the data records into a table or list. These data records may contain order information of the e-commerce platform, such as order number, order status, product name, purchase quantity, etc.

[0142] The encapsulated query results are embedded in an interface component, a container for displaying data in a user interface. An interface component can be a web page element, a view component in a mobile app, or a window component in a desktop application. By encapsulating query results in an interface component, you can ensure that the data is presented to users in a user-friendly and easy-to-understand manner. For example, on an e-commerce platform, query results can be encapsulated in a table on an order management page, where users can view and manage their order information.

[0143] Step S5200: The computer device embeds the interface component into the order management page of the e-commerce platform for display.

[0144] In practice, the order management page of an e-commerce platform typically includes multiple functional modules, such as order list, order details, and order status. By embedding query results into the order management page, users can conveniently view and manage their order information. For example, users can quickly find a specific order in the order list, view its details, or perform actions on it, such as canceling the order or confirming receipt.

[0145] The order management page is the interface through which users interact with the platform. Users can view and manage their order information through this page. By embedding query results into the order management page, users can conveniently view and manage their order information. For example, users can quickly find a specific order in the order list, view its details, or perform actions on it, such as canceling it or confirming receipt.

[0146] Specifically, the computer device can embed the interface component into the order management page in a variety of ways. For example, if the order management page is a webpage, the computer device can use HTML and JavaScript to dynamically embed the query results into a specific area of the webpage. If the order management page is a mobile application, the computer device can use a corresponding mobile development framework to embed the query results into the application's view component. Regardless of the method, the embedding process ensures that the query results are presented to the user in a user-friendly and easy-to-understand manner.

[0147] This embodiment achieves efficient display of query results and optimization of user interaction by pushing query results to the computer device that submits the data query statement, and the device encapsulates the query results in a preset format and embeds them into the interface component. This process not only ensures that the query results can be returned to the requester in a timely and accurate manner, but also displays the data on the order management page of the e-commerce platform in a user-friendly and easy-to-understand manner through encapsulation and embedding operations. This mechanism significantly improves the user experience, allowing users to quickly view and manage their order information, while improving operational efficiency. In this way, the e-commerce platform can better meet the user's management and operation needs for order information, and enhance the interactivity between the user and the platform. In addition, this encapsulation and embedding method also improves the flexibility and scalability of the data query system, allowing the system to adapt to different application scenarios and user needs, further improving the overall performance and user experience.

[0148] See also Figure 3According to one aspect of the present application, a data limited query control device is provided, comprising an interception analysis module 3100, a type determination module 3200, an engine query module 3300, and a proxy query module 3400. The interception analysis module 3100 is configured to intercept a data query statement, call a distributed data engine to generate a logical plan corresponding to the data query statement to be executed, and specifically execute a physical plan corresponding to the logical plan in a distributed file system; the type determination module 3200 is configured to determine whether the tree-like grammatical structure information of the data query statement contained in the logical plan matches a preset limited quick query special query. The engine query module 3300 is configured to call the distributed data engine to execute the physical plan when the data query statement does not belong to the limited quick search type, and obtain the query result corresponding to the data query statement in the distributed file system by executing the physical plan; the proxy query module 3400 is configured to determine the target data file in the distributed file system that matches the data query statement according to the file path in the physical plan when the data query statement belongs to the limited quick search type, and obtain the corresponding query result by executing the data query statement on the target data file.

[0149] Based on any embodiment of the device of the present application, the interception analysis module 3100 includes: an lazy execution module, configured to submit the data query statement to the distributed data engine for lazy execution to trigger the generation of the logical plan and physical plan, but not execute the physical plan; a logical acquisition module, configured to call a logical plan acquisition interface to obtain the logical plan, the logical plan includes tree-like grammatical structure information, the tree-like grammatical structure information decomposes the data query statement into multiple execution nodes, and defines the dependency relationship between each execution node, wherein at least one execution node includes the table name of the target data table that the data query statement needs to operate accordingly; a physical acquisition module, configured to call a physical plan acquisition interface to obtain the physical plan, the physical plan includes the file path of the target data file in the distributed file system determined according to the table name of the target data table.

[0150] Based on any embodiment of the device of the present application, the type determination module 3200 includes: a pre-judgment module, configured to determine whether the top-level execution node in the tree-like grammatical structure information of the logical plan matches the first limited quick-check feature, and when it does not match the first limited quick-check feature, determine that the data query statement does not belong to the limited quick-check type, and the first limited quick-check feature is used to limit the number of data queries required for the data query statement; an intermediate judgment module, configured to determine whether there is an execution node matching the second limited quick-check feature in the tree-like grammatical structure information when the first limited quick-check feature is matched, and when the execution node exists, determine that the data query statement does not belong to the limited quick-check type, and the second limited quick-check feature is used to indicate that the query complexity of the data query statement exceeds the expected condition; a hit setting module, configured to determine that the data query statement belongs to the limited quick-check type when there is no execution node matching the second limited quick-check feature.

[0151] Based on any embodiment of the device of the present application, the expected conditions indicated by the second limited quick search feature include any one or more of the following: the corresponding execution node indicates that the query scope of the data query statement exceeds a single data table; the corresponding execution node contains more than a preset number of filter conditions; the corresponding execution node contains nested subqueries; the corresponding execution node contains aggregation operations; the corresponding execution node contains window functions; the corresponding execution node contains projection operations across data tables.

[0152] Based on any embodiment of the device of the present application, the proxy query module 3400 includes: a path determination module, configured to extract the file path of the target data file corresponding to the data query statement from the physical plan; a system connection module, configured to connect to the distributed file system through the distributed file system client according to the file path, and locate the target data file; a record screening module, configured to read the data in the target data file, and screen and extract data records that meet the conditions according to the data query statement; and a result return module, configured to return the screened and extracted data records as query results.

[0153] Based on any embodiment of the device of the present application, the proxy query module 3400 includes: a data cache module, which is configured to associate the data query statement, the query result and the current timestamp corresponding to the query result, and store them in a cache system; a subsequent response module, which is configured to first execute a query from the cache system for a subsequently intercepted data query statement to determine the query result and its timestamp associated with the same data query statement; a synchronous multiplexing module, which is configured to determine whether the timestamp queried from the cache system exceeds a preset period, and when it does not exceed the preset period, respond to the subsequently intercepted data query statement with a corresponding query result.

[0154] Based on any embodiment of the device of the present application, the proxy query module 3400 includes: a data encapsulation module, configured to push the query results to the computer device that submits the data query statement, and the computer device encapsulates the query results in a preset format, and encapsulates each data record in the query results in an interface component, wherein the data record includes the order information of the e-commerce platform; a data display module, configured to embed the interface component into the order management page of the e-commerce platform for display by the computer device.

[0155] Another embodiment of the present application also provides a data limit query control device. Figure 4 Figure 2 shows a schematic diagram of the internal structure of a data limit query control device. The device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The non-volatile computer-readable storage medium of the device stores an operating system, a database, and computer-readable instructions. The database may store information sequences. When executed by the processor, the computer-readable instructions cause the processor to implement a data limit query control method.

[0156] The processor of the data limit query control device is used to provide computing and control capabilities, supporting the operation of the entire data limit query control device. The memory of the data limit query control device may store computer-readable instructions, which, when executed by the processor, can cause the processor to perform the data limit query control method of the present application. The network interface of the data limit query control device is used to connect and communicate with the terminal.

[0157] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the data limit query control device to which the scheme of the present application is applied. The specific data limit query control device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0158] In this embodiment, the processor is used to execute Figure 3 The memory stores the program code and various data required to execute the modules or submodules described above. The network interface is used to enable data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the data limited query control device of this application. The server can call the server's program code and data to execute the functions of all modules.

[0159] The present application also provides a non-volatile readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the data limited query control method of any embodiment of the present application.

[0160] The present application also provides a computer program product, comprising a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0161] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0162] To sum up, this application, with its unique technical principles and innovative processing flow, has demonstrated significant beneficial effects in improving query response speed and optimizing resource utilization efficiency, bringing new technical advantages to the field of distributed data query, and is particularly suitable for application scenarios such as e-commerce platforms and online live broadcast platforms.

Claims

1. A data limited query control method, characterized in that: include: Intercepting a data query statement, calling a distributed data engine to generate a logical plan corresponding to the data query statement to be executed, and specifically executing a physical plan corresponding to the logical plan in the distributed file system; determining whether the data query statement belongs to a limited quick search type according to whether the tree-like grammatical structure information of the data query statement contained in the logical plan matches a preset limited quick search feature; When the query does not belong to the limited quick query type, calling the distributed data engine to execute the physical plan, and obtaining the query result corresponding to the data query statement in the distributed file system by executing the physical plan; When it belongs to the limited quick search type, the target data file matching the data query statement in the distributed file system is determined according to the file path in the physical plan, and the corresponding query result is obtained by executing the data query statement on the target data file.

2. The data limited query control method according to claim 1, characterized in that: Invoking a distributed data engine to generate a logical plan corresponding to the data query statement to be executed, and specifically executing a physical plan corresponding to the logical plan in the distributed file system, including: Submitting the data query statement to the distributed data engine for lazy execution to trigger generation of the logical plan and the physical plan, but not executing the physical plan; Calling a logical plan acquisition interface to obtain the logical plan, wherein the logical plan includes tree-like grammatical structure information, the tree-like grammatical structure information decomposing the data query statement into multiple execution nodes and defining dependencies between the execution nodes, wherein at least one execution node includes the name of a target data table to be operated on by the data query statement; A physical plan acquisition interface is called to acquire the physical plan, where the physical plan includes a file path of a target data file in the distributed file system determined according to a table name of the target data table.

3. The data limited query control method according to claim 2, characterized in that: Determining whether the data query statement belongs to the limited quick search type according to whether the tree-like grammatical structure information of the data query statement included in the logical plan matches a preset limited quick search feature includes: determining whether a top-level execution node in the tree-like grammatical structure information of the logical plan matches a first limited quick-check feature; if the first limited quick-check feature is not matched, determining that the data query statement does not belong to a limited quick-check type, wherein the first limited quick-check feature is used to limit the number of data queries required by the data query statement; When the first limited quick search feature is matched, determining whether there is an execution node matching the second limited quick search feature in the tree-like grammatical structure information; if the execution node exists, determining that the data query statement does not belong to the limited quick search type, and the second limited quick search feature is used to indicate that the query complexity of the data query statement exceeds the expected condition; When there is no execution node matching the second limited quick search feature, it is determined that the data query statement belongs to the limited quick search type.

4. The data limited query control method according to claim 3, characterized in that: The expected conditions indicated by the second limited quick search feature include any one or more of the following: The corresponding execution node indicates that the query scope of the data query statement exceeds a single data table; The corresponding execution node contains more than the preset number of filter conditions; The corresponding execution node contains nested subqueries; The corresponding execution node contains the aggregation operation; The corresponding execution node contains the window function; The corresponding execution node contains projection operations across data tables.

5. The data limited query control method according to claim 1, characterized in that: Determining a target data file in the distributed file system that matches the data query statement according to the file path in the physical plan, and obtaining a corresponding query result by executing the data query statement on the target data file, including: Extracting a file path of a target data file corresponding to the data query statement from the physical plan; Connecting to the distributed file system through the distributed file system client according to the file path and locating the target data file; Read the data in the target data file, and filter and extract data records that meet the conditions according to the data query statement; Return the filtered and extracted data records as query results.

6. The data limited query control method according to any one of claims 1 to 5, characterized in that: After obtaining the corresponding query result by executing the data query statement on the target data file, the method includes: Associating the data query statement, the query result, and a current timestamp corresponding to obtaining the query result, and storing them in a cache system; For subsequently intercepted data query statements, a query is first executed from the cache system to determine query results and their timestamps associated with the same data query statement; It is determined whether the timestamp retrieved from the cache system exceeds a preset period, and when it does not exceed the preset period, the subsequently intercepted data query statement is responded to with a corresponding query result.

7. The data limited query control method according to any one of claims 1 to 5, characterized in that: After obtaining the corresponding query result by executing the data query statement on the target data file, the method includes: Pushing the query results to the computer device that submitted the data query statement, which encapsulates each data record in the query results in a preset format into an interface component, wherein the data record includes order information from the e-commerce platform; The computer device embeds the interface component into the order management page of the e-commerce platform for display.

8. A data limited query control device, characterized in that: include: An interception and analysis module is configured to intercept a data query statement, call a distributed data engine to generate a logical plan corresponding to the data query statement to be executed, and specifically execute a physical plan corresponding to the logical plan in the distributed file system; a type determination module configured to determine whether the data query statement belongs to a limited quick search type according to whether the tree grammatical structure information of the data query statement included in the logical plan matches a preset limited quick search feature; An engine query module configured to, when the query does not belong to a limited quick query type, call the distributed data engine to execute the physical plan, and obtain a query result corresponding to the data query statement in the distributed file system by executing the physical plan; The proxy query module is configured to determine, when belonging to a limited quick search type, a target data file in the distributed file system that matches the data query statement based on the file path in the physical plan, and obtain a corresponding query result by executing the data query statement on the target data file.

9. A data limited query control device, comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images