A data query method, device and electronic equipment of a Spark engine

Abstract

The application provides a data query method and device of a Spark engine and electronic equipment, and relates to the technical field of data processing. In the application, in response to a data query request initiated by a target object, a task session is created for the target object; after it is determined that the task session is created, the data query request is parsed to determine a data query mode and a data volume of to-be-queried data; an initial configuration strategy is adjusted based on the data query mode and the data volume to obtain a target configuration strategy; wherein the initial configuration strategy comprises initial component configurations respectively set for each component included in the Spark engine; and the target configuration strategy is sent to the Spark engine through the task session, so that the Spark engine queries the to-be-queried data from a fixed storage space where the to-be-queried data is located based on the target configuration strategy. In this way, the data query performance of the Spark engine can better meet different data query requirements.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data query method, apparatus and electronic device for the Spark engine. Background Technology

[0002] In Apache Spark SQL, the coordinated operation of various components within the Spark engine (such as the cost-based optimizer (CBO), adaptive query execution (AQE), dynamic partition pruning (DPP), and Shuffle) enables the querying of various business data (e.g., financial business data or operational data). However, in these data querying methods, the typically fixed configurations of each component can make it difficult for the Spark engine's data query performance to meet diverse data query needs. Therefore, ensuring that the Spark engine's data query performance meets these varied data query requirements is a pressing issue that needs to be addressed. Summary of the Invention

[0003] This application provides a data query method, apparatus, and electronic device for Spark engine to ensure that the data query performance of Spark engine can better meet different data query needs.

[0004] In a first aspect, embodiments of this application provide a data query method for the Spark engine, the method comprising: In response to a data query request initiated by the target object, a task session is created for the target object; After confirming that the task session has been created, the data query request is parsed to determine the data query mode and data volume of the data to be queried. The initial configuration strategy is adjusted based on the data query pattern and data volume to obtain the target configuration strategy; the initial configuration strategy includes: initial component configurations set for each component included in the Spark engine; The target configuration strategy is sent to the Spark engine through the task session, so that the Spark engine can query the data to be queried from the fixed storage space where the data to be queried is located based on the target configuration strategy.

[0005] In one optional embodiment, the data query request is parsed to determine the data query pattern and data volume of the data to be queried, including: Send the data query request to the Hive engine; Receive Hive metadata returned by the Hive engine based on the data query request; Hive metadata includes at least: the data query mode and data volume of the data to be queried.

[0006] In one optional embodiment, the initial configuration strategy is adjusted based on the data query pattern and data volume to obtain the target configuration strategy, including: The queue length and resource expansion information of the task queue are determined based on the amount of data; the task queue is used to store multiple data query subtasks corresponding to a data query request. Based on queue length, resource expansion information, and at least one task execution rule associated with the data query pattern, the initial configuration strategy is adjusted to obtain the target configuration strategy.

[0007] In one optional embodiment, determining the queue length and resource expansion information of the task queue based on the data volume includes: The queue length of the task queue is determined based on the range of data volume, and the sub-data volume corresponding to each key in the data to be queried is determined from the data volume. Perform the following operations for each key: If the first sub-data volume corresponding to the first key is greater than the preset first data volume threshold, then based on the quantitative relationship between the first sub-data volume and the first data volume threshold, the first key is split into multiple sub-keys; where the first key is any one of the keys. Based on the amount of sub-data corresponding to each sub-key, partitions are set for each sub-key; If the first sub-data volume is less than the preset second data volume threshold, the partition set for the first key will be merged with the partition set for at least one second key included in each key; wherein the sub-data volume corresponding to each second key is less than the second data volume threshold, and the second data volume threshold is less than the first data volume threshold.

[0008] In an optional embodiment, the method further includes: Records the performance indicator parameter sets corresponding to the various levels of performance indicators during the Spark engine's query of data from a fixed storage space based on the target configuration strategy; Based on the mean and threshold values ​​of the performance index parameters corresponding to multiple performance index parameter sets, the performance index evaluation values ​​corresponding to the multi-level performance indexes are determined. Based on the performance evaluation values ​​and weights corresponding to the multi-level performance metrics, the query performance evaluation value of the Spark engine is obtained.

[0009] In an optional embodiment, after the target configuration policy is sent to the Spark engine via a task session, so that the Spark engine can query the data to be queried from the fixed storage space where the data to be queried resides based on the target configuration policy, the method further includes: In response to a data retrieval request from a target object for the data to be queried, based on at least one data element identifier carried in the data retrieval request, the data elements corresponding to the at least one data element identifier are retrieved from the data to be queried. Present at least one data element to the target object through a task session.

[0010] In an optional embodiment, after the target configuration policy is sent to the Spark engine via a task session, so that the Spark engine can query the data to be queried from the fixed storage space where the data to be queried resides based on the target configuration policy, the method further includes: If the number of queries for the data to be queried within a set time range exceeds a set threshold, a temporary cache space for the data to be queried will be constructed based on the business scenario corresponding to the data to be queried. The data to be queried is compressed according to the set data format, and the compressed data to be queried is saved in a temporary cache space.

[0011] In one alternative embodiment, the components include at least one or a combination of CBO, AQE, DPP, and Shuffle.

[0012] Secondly, embodiments of this application also provide a data query device for the Spark engine, the device comprising: The session creation module is used to create a task session for the target object in response to a data query request initiated by the target object. The request parsing module is used to parse the data query request after the task session has been created, and to determine the data query mode and data volume of the data to be queried. The strategy adjustment module is used to adjust the initial configuration strategy based on the data query pattern and data volume to obtain the target configuration strategy; the initial configuration strategy includes: initial component configurations set for each component included in the Spark engine; The data query module is used to send the target configuration strategy to the Spark engine through the task session, so that the Spark engine can query the data to be queried from the fixed storage space where the data to be queried is located based on the target configuration strategy.

[0013] In one optional embodiment, when parsing a data query request to determine the data query pattern and data volume of the data to be queried, the request parsing module is specifically used for: Send the data query request to the Hive engine; Receive Hive metadata returned by the Hive engine based on the data query request; Hive metadata includes at least: the data query mode and data volume of the data to be queried.

[0014] In an optional embodiment, when adjusting the initial configuration strategy based on the data query pattern and data volume to obtain the target configuration strategy, the strategy adjustment module is specifically used for: The queue length and resource expansion information of the task queue are determined based on the amount of data; the task queue is used to store multiple data query subtasks corresponding to a data query request. Based on queue length, resource expansion information, and at least one task execution rule associated with the data query pattern, the initial configuration strategy is adjusted to obtain the target configuration strategy.

[0015] In an optional embodiment, when determining the queue length and resource expansion information of the task queue based on the data volume, the policy adjustment module is specifically used for: The queue length of the task queue is determined based on the range of data volume, and the sub-data volume corresponding to each key in the data to be queried is determined from the data volume. Perform the following operations for each key: If the first sub-data volume corresponding to the first key is greater than the preset first data volume threshold, then based on the quantitative relationship between the first sub-data volume and the first data volume threshold, the first key is split into multiple sub-keys; where the first key is any one of the keys. Based on the amount of sub-data corresponding to each sub-key, partitions are set for each sub-key; If the first sub-data volume is less than the preset second data volume threshold, the partition set for the first key will be merged with the partition set for at least one second key included in each key; wherein the sub-data volume corresponding to each second key is less than the second data volume threshold, and the second data volume threshold is less than the first data volume threshold.

[0016] In an optional embodiment, the data query module is further configured to: Records the performance indicator parameter sets corresponding to the various levels of performance indicators during the Spark engine's query of data from a fixed storage space based on the target configuration strategy; Based on the mean and threshold values ​​of the performance index parameters corresponding to multiple performance index parameter sets, the performance index evaluation values ​​corresponding to the multi-level performance indexes are determined. Based on the performance evaluation values ​​and weights corresponding to the multi-level performance metrics, the query performance evaluation value of the Spark engine is obtained.

[0017] In an optional embodiment, after the target configuration policy is sent to the Spark engine via a task session, enabling the Spark engine to query the data to be queried from the fixed storage space where the data to be queried resides based on the target configuration policy, the session creation module is further configured to: In response to a data retrieval request from a target object for the data to be queried, based on at least one data element identifier carried in the data retrieval request, the data elements corresponding to the at least one data element identifier are retrieved from the data to be queried. Present at least one data element to the target object through a task session.

[0018] In an optional embodiment, after the target configuration policy is sent to the Spark engine via a task session, enabling the Spark engine to query the data to be queried from the fixed storage space where the data to be queried resides based on the target configuration policy, the data query module is further configured to: If the number of queries for the data to be queried within a set time range exceeds a set threshold, a temporary cache space for the data to be queried will be constructed based on the business scenario corresponding to the data to be queried. The data to be queried is compressed according to the set data format, and the compressed data to be queried is saved in a temporary cache space.

[0019] Thirdly, embodiments of this application provide an electronic device, including: processor; Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform data querying methods of the Spark engine as described in the first aspect.

[0020] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the data query method of the Spark engine as described in the first aspect.

[0021] Fifthly, this application provides a computer program product that, when invoked by a computer, causes the computer to execute the data query method steps of the Spark engine as described in the first aspect.

[0022] The beneficial effects of this application are as follows: In the Spark engine data query method provided in this application embodiment, in response to a data query request initiated by a target object, a task session is created for the target object. Then, after confirming the task session creation is complete, the data query request is parsed to determine the data query mode and data volume of the data to be queried. Further, the initial configuration strategy is adjusted based on the data query mode and data volume to obtain a target configuration strategy. The initial configuration strategy includes initial component configurations set for each component included in the Spark engine. Finally, the target configuration strategy is sent to the Spark engine through the task session, enabling the Spark engine to query the data to be queried from the fixed storage space where the data to be queried resides based on the target configuration strategy. Therefore, by designing the Spark engine's target configuration strategy specifically according to the data query mode and data volume of the data to be queried, the resources configured for the data query request (or the data to be queried) are highly matched with the actual resource configuration required for querying the data to be queried, reducing the problem of resource waste or insufficiency, thereby ensuring that the Spark engine's data query performance can better meet different data query needs.

[0023] Furthermore, other features and advantages of this application will be set forth in the following description and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described herein are used to provide a further understanding of this application, constitute a part of this application, and do not constitute an improper limitation of this application. In the accompanying drawings: Figure 1 This is a schematic diagram of an optional system architecture applicable to the embodiments of this application.

[0025] Figure 2 This is a schematic diagram illustrating the implementation process of a data query method using the Spark engine, as provided in an embodiment of this application.

[0026] Figure 3 This is a logical diagram illustrating a target configuration generation strategy provided in an embodiment of this application.

[0027] Figure 4 This is a schematic diagram illustrating a data query scenario provided in an embodiment of this application.

[0028] Figure 5 A method based on the embodiments of this application is provided. Figure 2 The logic diagram.

[0029] Figure 6 This is a schematic diagram of the structure of a data query device for the Spark engine provided in an embodiment of this application.

[0030] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0032] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0033] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0034] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0035] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0036] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0037] (1) Kyuubi: A multi-tenant SQL gateway and engine management layer for Spark, supporting session-level / engine-level parameter configuration and isolation.

[0038] (2) AQE: As a dynamic optimizer, after the Shuffle phase is completed, AQE uses the precise statistical information it generates to dynamically adjust subsequent plans that have not yet been executed. AQE can be used to dynamically adjust Shuffle partitions, Join strategies, skew splits, etc. based on runtime statistics.

[0039] (3) CBO: As a static optimizer, CBO can evaluate the cost of different execution plans based on the table's statistical information (such as data volume and data distribution) before query execution and select the plan with the lowest cost. In other words, CBO can be used to select the optimal execution plan based on data statistical information.

[0040] (4) Shuffle: This is a key stage in Spark distributed computing, involving network transmission and repartitioning of data. The intermediate results of Shuffle (such as data file size and partition size) are the basis for subsequent dynamic optimization.

[0041] (5) DPP: In the Join operation, the filter conditions are dynamically transmitted to the fact table using the dimension table data that has been filtered at runtime. Only the relevant partitions are scanned, reducing input / output (I / O). It relies on equal join and broadcast threshold, which complements the optimization timing of AQE (after Shuffle) to jointly reduce the amount of data scanned, that is, reduce the scan of irrelevant partitions.

[0042] (6) Data skew governance: split or sample and redistribute extreme hotkey / long tail distributions to reduce the long tail.

[0043] (7) Channel Attribution: In multi-touchpoint marketing and risk control analysis, calculate the contribution of different channels / paths to target conversion.

[0044] (8) Wide table for tracking: flatten and aggregate multidimensional features, behavioral events, etc. to form an analysis table with a high number of columns and wide fields.

[0045] Based on the above explanations of terms and related terminology, the design concept of the embodiments of this application will be briefly introduced below: In Apache Spark SQL, the coordinated work of various components within the Spark engine (such as CBO, AQE, DPP, and Shuffle) enables the querying of various business data (e.g., financial business data or operational data). Among these, queries using frequently used data tracking and channel attribution wide tables often involve multi-table joins, window aggregations, complex filtering, uneven data distribution, and frequent statistical updates. However, in these data query methods, the typically fixed configurations of each component can make it difficult for the Spark engine's query performance to meet diverse data query needs.

[0046] In view of this, in order to solve or improve the above problems, embodiments of this application provide a database capacity assessment method, which may specifically include: responding to a data query request initiated by a target object, creating a task session for the target object; then, after determining that the task session has been created, parsing the data query request to determine the data query mode and data volume of the data to be queried; further, adjusting the initial configuration strategy based on the data query mode and data volume to obtain a target configuration strategy; wherein, the initial configuration strategy includes: initial component configurations set separately for each component included in the Spark engine; finally, sending the target configuration strategy to the Spark engine through the task session, so that the Spark engine can query the data to be queried from the fixed storage space where the data to be queried is located based on the target configuration strategy.

[0047] In particular, the preferred embodiments of this application will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other unless otherwise specified.

[0048] See Figure 1The diagram illustrates an optional system architecture applicable to an embodiment of this application. This system architecture may include: terminal devices (101a, 101b) and server 102. The terminal devices (101a, 101b) and server 102 can interact via a communication network. The communication network may employ wireless communication or wired communication methods. For example, the terminal devices (101a, 101b) can access the network and communicate with server 102 via cellular mobile communication technology. The aforementioned cellular mobile communication technology may include, for example, 5th generation mobile networks (5G) technology or next-generation mobile communication technology. Optionally, the terminal devices (101a, 101b) can access the network and communicate with server 102 via short-range wireless communication. The aforementioned short-range wireless communication method may include, for example, wireless fidelity (Wi-Fi) technology.

[0049] This application embodiment does not impose any limitation on the number of communication devices involved in the above system architecture. For example, the above system architecture may include more terminal devices, or fewer terminal devices, or other network devices. Figure 1 As shown, only terminal devices (101a, 101b) and server 102 are described as examples. The following is a brief introduction to each of the above communication devices and their respective functions.

[0050] A terminal device (101a, 101b) is a device that can provide voice and / or data connectivity to a user, and may be a device that supports wired and / or wireless connections.

[0051] For example, terminal devices (101a, 101b) may include, but are not limited to: mobile phones, tablets, laptops, handheld computers, mobile internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminal devices in industrial control, wireless terminal devices in autonomous driving, wireless terminal devices in smart grids, wireless terminal devices in transportation safety, wireless terminal devices in smart cities, or wireless terminal devices in smart homes, etc.

[0052] In addition, the terminal devices (101a, 101b) can have related clients installed. These clients can be software, such as applications (APPs), browsers, short video software, web pages, mini programs, etc.

[0053] It should be noted that the terminal devices (101a, 101b) in this application embodiment can enable the aforementioned client related to data querying of the Spark engine to send a data query request initiated by the target object to the server 102, so as to carry out subsequent methods and steps such as data querying of the Spark engine for the data to be queried.

[0054] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0055] It is worth noting that, in this embodiment, server 102 can be used to respond to a data query request initiated by a target object, create a task session for the target object; after determining that the task session has been created, parse the data query request to determine the data query mode and data volume of the data to be queried; adjust the initial configuration strategy based on the data query mode and data volume to obtain a target configuration strategy; wherein, the initial configuration strategy includes: initial component configurations set for each component included in the Spark engine; send the target configuration strategy to the Spark engine through the task session, so that the Spark engine can query the data to be queried from the fixed storage space where the data to be queried is located based on the target configuration strategy. Figure 1 As shown, server 102 can perform data queries in response to data query requests through the data query system.

[0056] The data query method of the Spark engine provided by the exemplary embodiments of this application is described below in conjunction with the above system architecture and with reference to the accompanying drawings. It should be noted that the above system architecture is only shown for the purpose of understanding the spirit and principles of this application, and the embodiments of this application are not limited in any way.

[0057] See Figure 2 The diagram illustrates the implementation flow of a data query method using the Spark engine, as provided in this embodiment. Taking a server as an example, the specific implementation flow of this method is as follows: S201: In response to a data query request initiated by the target object, create a task session for the target object.

[0058] The target object mentioned above can be a user or a tenant, etc. The data query request mentioned above can be initiated by the target object from a terminal device (such as a client or a platform). Furthermore, the data query request mentioned above can be for data of various types and in various business scenarios, such as credit data in a financial scenario. This application embodiment does not specifically limit this.

[0059] In one alternative implementation, during step S201, after receiving a data query request initiated by the target object, the server can create a separate task session for the target object via Kyuubi.

[0060] This avoids multiple tasks existing in the same session, which could cause interference between other tasks and the data query task. Therefore, it achieves a higher level of isolation and control over data query modes, resources, and parameters.

[0061] S202: After confirming that the task session has been created, parse the data query request to determine the data query mode and data volume of the data to be queried.

[0062] Optionally, the data query mode described above can be SQL mode, which can include strict mode and lenient mode. Strict mode executes the task by immediately reporting an error and terminating the current SQL task if the SQL operation violates data rules (e.g., inserting null values ​​into non-null fields, incorrect date formats). Lenient mode executes the task by automatically "correcting" the data when it is invalid (e.g., truncating excessively long strings, filling invalid dates with 0000-00-00), only outputting a warning, and not terminating the current SQL task.

[0063] The aforementioned data volume can include the sub-data volume corresponding to each key in the data to be queried. Each key can be the name of one or more data elements within the data to be queried, and each element name (i.e., each key) can correspond to one or more data elements. In other words, the aforementioned data volume not only reflects the total quantity of the data to be queried but also the data volume corresponding to each type of data element within the data to be queried.

[0064] In one optional implementation, during step S202, the server can send a data query request to the Hive engine, thereby receiving Hive metadata returned by the Hive engine based on the data query request. The Hive metadata may include at least the data query mode and data volume of the data to be queried. Using this method, the Hive engine can accurately determine the data query module and data volume of the data to be queried, thus ensuring the successful execution of the data query.

[0065] S203: Adjust the initial configuration strategy based on the data query pattern and data volume to obtain the target configuration strategy.

[0066] The initial configuration strategy may include initial component configurations set separately for each component included in the Spark engine. These components may include at least one or a combination of CBO, AQE, DPP, and Shuffle.

[0067] It's important to note that CBO generates the initial execution plan, and Shuffle is a crucial data exchange stage during execution. AQE, after Shuffle completes, uses the runtime data it generates to dynamically optimize subsequent execution stages. DPP optimizes during the data reading stage. CBO, AQE, DPP, and Shuffle together form a complete chain in Spark, from static planning to runtime adaptive optimization, significantly improving query efficiency and stability.

[0068] In one alternative implementation, see [link to relevant documentation]. Figure 3 As shown, when executing step S203, the server can determine the queue length and resource expansion information of the task queue based on the amount of data to be queried. Then, based on the queue length, resource expansion information, and at least one task execution rule associated with the data query mode, the server can adjust the initial configuration strategy (i.e., the initial component configuration set by each component) to obtain the target configuration strategy.

[0069] The task queue mentioned above can be used to store multiple data query subtasks corresponding to a data query request. The aforementioned multiple data query subtasks are multiple subtasks that need to be executed to complete the query of the aforementioned data to be queried.

[0070] The resource scaling information mentioned above can be horizontal scaling information. For example, it can include scaling or configuration information set separately for partitions, replicas, and load balancing. It's worth noting that partitioning can divide a task queue (or message queue) into multiple logical partitions, each capable of independently processing tasks, thus achieving parallel processing and horizontal scaling. Replicating data across multiple nodes ensures that data is not lost if a node fails and can be recovered from other nodes, thereby ensuring high availability and data security for the task queue. Load balancing is a crucial means of ensuring system performance and stability. It evenly distributes the various data query subtasks in the task queue across different nodes, preventing some nodes from being overloaded while others are idle.

[0071] Furthermore, the above-mentioned task execution rules may include, but are not limited to: how to verify data validity, whether to be compatible with different SQL standards, whether to report errors directly or to "handle" errors leniently, and task execution rules such as grouped queries and string comparisons. This application embodiment does not specifically limit these rules.

[0072] In one alternative implementation, when determining the queue length and resource expansion information of the task queue based on the amount of data to be queried, the server can determine the queue length based on the quantity range to which the amount of data to be queried belongs. In this way, based on the mapping relationship between quantity ranges and queue lengths, the queue length of the task queue can be quickly determined after the amount of data to be queried is determined, thereby improving the efficiency of data querying.

[0073] Understandably, the queue length can be determined based on the range of data volume within a given interval. Optionally, a larger data volume range results in a longer queue, while a smaller data volume range results in a shorter queue. Therefore, the queue length can be dynamically adjusted based on the amount of data to be queried.

[0074] Furthermore, the server can also determine the sub-data volume corresponding to each key within the query data from the total data volume of the query data. The sum of the sub-data volumes corresponding to each key is the total data volume of the query data. Then, for any one of the keys, such as the first key, the following operations can be performed: If the amount of the first sub-data corresponding to the first key is greater than a preset first data volume threshold, the server can split the first key into multiple sub-keys based on the quantitative relationship between the first sub-data volume and the preset first data volume threshold, thus achieving hotkey splitting. In other words, when the amount of sub-data is greater than the preset first data volume threshold, the corresponding key can be determined to be a hotkey. Optionally, the aforementioned quantitative relationship can be: the first sub-data volume is the rounded-up value of a multiple of the preset first data volume threshold.

[0075] For example, if the amount of the first sub-data corresponding to the first key is 2.5 times the first data volume threshold, the server can split the first key into 3 sub-keys. As another example, if the amount of the first sub-data corresponding to the first key is 3.2 times the first data volume threshold, the server can split the first key into 4 sub-keys.

[0076] When the server splits the first key into multiple subkeys, it can divide the first key into subkeys based on the element similarity between the multiple data elements corresponding to the first key. Therefore, the number of elements corresponding to different subkeys may be the same or different, and this embodiment does not specifically limit this.

[0077] Furthermore, after splitting the primary key into multiple subkeys, the server can then set up partitions for each subkey based on the amount of data corresponding to each subkey. This approach, which partitions different types of data elements separately, not only improves the efficiency of data retrieval but also enhances data security.

[0078] If the amount of first sub-data corresponding to the first key is less than the preset second data volume threshold, the server can merge the partition set for the first key with the partitions set for at least one second key included in each key. In this way, partitions with smaller data volume thresholds can be merged, reducing the number of partitions.

[0079] In this case, the amount of sub-data corresponding to each second key is less than the preset second data amount threshold, and the preset second data amount threshold is less than the preset first data amount threshold.

[0080] S204: Send the target configuration policy to the Spark engine through the task session so that the Spark engine can query the data to be queried from the fixed storage space where the data to be queried is located based on the target configuration policy.

[0081] The Spark engine is a fast and general-purpose computing engine designed specifically for large-scale data processing. The Spark engine can include multiple worker nodes, which are used to complete data queries on the data to be queried.

[0082] In an alternative implementation, the server can also record the performance metric parameter sets corresponding to the multi-level performance metrics during the Spark engine's query of data from a fixed storage space based on the target configuration strategy.

[0083] The multi-level performance metrics can include latency, data volume, skewness, retry and failure reasons, etc., corresponding to the SQL level, Stage level, and Task level, respectively. A single SQL statement can trigger one or more Jobs, each Job can be divided into one or more Stages, and each Stage can be divided into one or more Tasks.

[0084] Next, the server can determine the performance evaluation values ​​corresponding to the multi-level performance indicators based on the average performance indicator parameters and the threshold values ​​of the performance indicator parameters corresponding to the multiple performance indicator parameter sets.

[0085] Optionally, the performance evaluation value of each performance indicator can be determined by the ratio of the difference between the mean of the performance indicator parameters and the threshold of the corresponding performance indicator parameter set to the threshold of the performance indicator parameter.

[0086] Ultimately, the server can obtain the query performance evaluation value of the Spark engine based on the performance evaluation values ​​and weights corresponding to the multi-level performance metrics. In other words, the query performance evaluation value of the Spark engine can be obtained by weighted summation of the performance evaluation values ​​corresponding to the multi-level performance metrics.

[0087] Based on the above approach, by collecting the performance indicator parameter sets corresponding to each of the multi-level performance indicators, accurate gray-scale verification and rollback can be achieved. In other words, by collecting the performance indicator parameter sets corresponding to each of the multi-level performance indicators, parameter adjustment and rule rewriting can be realized, thereby supporting gray-scale verification and rapid rollback.

[0088] In one alternative implementation, see [link to relevant documentation]. Figure 4 As shown, after the Spark engine queries the data to be queried from the fixed storage space where the data to be queried resides based on the target configuration strategy, the server can respond to the target object's data retrieval request for the data to be queried. Based on at least one data element identifier (e.g., data element identifier 1 to data element identifier N) carried in the data retrieval request, it retrieves the data elements (e.g., data element 1 to data element N) corresponding to each of the at least one data element identifier from the data to be queried, and then displays at least one data element to the target object through the task session. In this way, the target object can clearly view a portion of the data in the data to be queried through the task session.

[0089] Queries involving frequently used business data (such as event tracking and channel attribution wide tables) often involve multi-table joins, window aggregations, and complex filtering, resulting in uneven data distribution and frequent statistical updates. Therefore, to reduce repetitive data queries on frequently used business data and save on query overhead, one optional implementation involves the server retrieving the data from a fixed storage space based on the target configuration strategy. If the number of queries for the data within a set time range exceeds a set threshold, a temporary cache space for the data is then constructed based on the corresponding business scenario.

[0090] For example, assuming the aforementioned time range is 30 minutes and the aforementioned query threshold is 3, if the number of queries for the data to be queried within 30 minutes is 6 (greater than 3), the server can build a temporary cache space for the data based on the business scenario corresponding to the data. Conversely, if the number of queries for the data to be queried within 30 minutes is 2 (less than 3), the server does not need to build a temporary cache space for the data. It should be understood that after the server builds a temporary cache space for the data to be queried, it can release the temporary cache space once it determines that the number of queries for the data within the set time range is less than or equal to the set query threshold.

[0091] Next, after constructing a temporary cache space for the data to be queried, the server can compress the data according to the set data format and save the compressed data in the temporary cache space. Optionally, the temporary cache form of the data to be queried can be a temporary table, etc., which is not specifically limited in this embodiment.

[0092] This approach not only allows for the creation of temporary storage space for frequently queried business data, reducing the overhead of multiple repetitive data queries, but also enables the compression of the data to be queried before storing it in the temporary storage space, thereby reducing the storage space required to build the temporary storage space.

[0093] Based on the data query methods for the Spark engine described in steps S201-S204 above, please refer to... Figure 5 The diagram illustrates a logical representation of a data query using the Spark engine, as provided in this embodiment. The data query system corresponding to the server may include Kyubi, a resource orchestrator, the Spark engine, and an observation and governance module. The target object can send a data query request to Kyubi via a client / platform. Upon receiving the request, Kyubi creates a task session for the target object. The resource orchestrator (e.g., Yarn) sends the session content to the Hive engine to obtain Hive metadata, thereby determining the data query mode and data volume based on the metadata. Further, the resource orchestrator adjusts the initial configuration strategy based on the query mode and data volume to obtain a target configuration strategy, which is then sent to the Spark engine (or Spark cluster). Upon receiving the target configuration strategy, the Spark engine adjusts the component configurations of various components (e.g., CBO, AQE, DPP, and Shuffle) and the node configurations of each worker node to read the data to be queried stored in the storage layer. Figure 5As shown, various metrics from Kyubi, the resource orchestrator, and the Spark engine during the execution of queries on the data to be queried can be sent to the observation and governance module to achieve closed-loop governance. Furthermore, various metrics and target configuration strategies from the Spark engine during the execution of queries on the data to be queried can also be fed back to Kyubi through the resource orchestrator for the target object to view.

[0094] It should be noted that, based on the above-mentioned data query method using Spark pins, resource allocation and ultra-large data query tasks are supported. For example, this may include: setting independent queues and concurrency limits for ultra-large jobs to avoid mutual interference; setting executorMemory, memoryOverhead, and cores according to job profiles; configuring external shuffle, parallel write, and spill parameters to stabilize high traffic; and using staged retries and speculative execution to reduce the impact of long-tail tasks.

[0095] Furthermore, the scenario-oriented session-level adaptive parameter template system optimizes and speeds up the execution plan for complex queries on the data to be queried (e.g., wide tables for data tracking, channel attribution, etc.) by optimizing queries and scheduling resources, leveraging Kyubi's multi-tenant session / engine and parameter injection capabilities, and combining Spark SQL's AQE, CBO, DPP, data skew governance, and broadcast / Shuffle adaptive strategies (i.e., target configuration strategies). This solves the problem of the lack of "adaptive" orchestration capabilities on the resource side, which leads to either resource waste or insufficient stability during peak / ultra-large jobs.

[0096] In summary, the data query method for the Spark engine provided in this application embodiment involves creating a task session for the target object in response to a data query request. Then, after the task session is confirmed to be complete, the data query request is parsed to determine the data query mode and data volume of the data to be queried. Further, the initial configuration strategy is adjusted based on the data query mode and data volume to obtain a target configuration strategy. The initial configuration strategy includes initial component configurations set for each component included in the Spark engine. Finally, the target configuration strategy is sent to the Spark engine through the task session, enabling the Spark engine to query the data to be queried from the fixed storage space where the data to be queried resides based on the target configuration strategy. Therefore, by designing the target configuration strategy for the Spark engine specifically according to the data query mode and data volume of the data to be queried, the resources configured for the data query request are highly matched with the actual resource configuration required to query the data, reducing resource waste or insufficiency and ensuring that the data query performance of the Spark engine can better meet different data query needs.

[0097] Furthermore, based on the same technical concept, this application also provides a data query device for the Spark engine, which is used to implement the above-described method flow of this application embodiment. For example, see [link to relevant documentation]. Figure 6 As shown, the data query device 600 of the Spark engine may include: a session creation module 601, a request parsing module 602, a strategy adjustment module 603, and a data query module 604, wherein: The session creation module 601 is used to create a task session for the target object in response to a data query request initiated by the target object. The request parsing module 602 is used to parse the data query request after the task session is determined to be created, and to determine the data query mode and data volume of the data to be queried. The strategy adjustment module 603 is used to adjust the initial configuration strategy based on the data query mode and data volume to obtain the target configuration strategy; wherein, the initial configuration strategy includes: initial component configurations set for each component included in the Spark engine; The data query module 604 is used to send the target configuration strategy to the Spark engine through the task session, so that the Spark engine can query the data to be queried from the fixed storage space where the data to be queried is located based on the target configuration strategy.

[0098] In an optional embodiment, when parsing a data query request to determine the data query pattern and data volume of the data to be queried, the request parsing module 602 is specifically used for: Send the data query request to the Hive engine; Receive Hive metadata returned by the Hive engine based on the data query request; Hive metadata includes at least: the data query mode and data volume of the data to be queried.

[0099] In an optional embodiment, when adjusting the initial configuration strategy based on the data query pattern and data volume to obtain the target configuration strategy, the strategy adjustment module 603 is specifically used for: The queue length and resource expansion information of the task queue are determined based on the amount of data; the task queue is used to store multiple data query subtasks corresponding to a data query request. Based on queue length, resource expansion information, and at least one task execution rule associated with the data query pattern, the initial configuration strategy is adjusted to obtain the target configuration strategy.

[0100] In an optional embodiment, when determining the queue length and resource expansion information of the task queue based on the amount of data, the policy adjustment module 603 is specifically used for: The queue length of the task queue is determined based on the range of data volume, and the sub-data volume corresponding to each key in the data to be queried is determined from the data volume. Perform the following operations for each key: If the first sub-data volume corresponding to the first key is greater than the preset first data volume threshold, then based on the quantitative relationship between the first sub-data volume and the first data volume threshold, the first key is split into multiple sub-keys; where the first key is any one of the keys. Based on the amount of sub-data corresponding to each sub-key, partitions are set for each sub-key; If the first sub-data volume is less than the preset second data volume threshold, the partition set for the first key will be merged with the partition set for at least one second key included in each key; wherein the sub-data volume corresponding to each second key is less than the second data volume threshold, and the second data volume threshold is less than the first data volume threshold.

[0101] In an optional embodiment, the data query module 604 is further configured to: Records the performance indicator parameter sets corresponding to the various levels of performance indicators during the Spark engine's query of data from a fixed storage space based on the target configuration strategy; Based on the mean and threshold values ​​of the performance index parameters corresponding to multiple performance index parameter sets, the performance index evaluation values ​​corresponding to the multi-level performance indexes are determined. Based on the performance evaluation values ​​and weights corresponding to the multi-level performance metrics, the query performance evaluation value of the Spark engine is obtained.

[0102] In an optional embodiment, after the target configuration policy is sent to the Spark engine via a task session, so that the Spark engine can query the data to be queried from the fixed storage space where the data to be queried resides based on the target configuration policy, the session creation module 601 is further configured to: In response to a data retrieval request from a target object for the data to be queried, based on at least one data element identifier carried in the data retrieval request, the data elements corresponding to the at least one data element identifier are retrieved from the data to be queried. Present at least one data element to the target object through a task session.

[0103] In an optional embodiment, after the target configuration policy is sent to the Spark engine via a task session, enabling the Spark engine to query the data to be queried from the fixed storage space where the data to be queried resides based on the target configuration policy, the data query module 604 is further configured to: If the number of queries for the data to be queried within a set time range exceeds a set threshold, a temporary cache space for the data to be queried will be constructed based on the business scenario corresponding to the data to be queried. The data to be queried is compressed according to the set data format, and the compressed data to be queried is saved in a temporary cache space.

[0104] Based on the description of the method and apparatus embodiments above, an exemplary embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method according to an embodiment of the present invention.

[0105] This application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0106] This application also provides a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0107] See Figure 7 The diagram shown below illustrates the structure of an electronic device 700 that can serve as a server or client in this application, and is an example of a hardware device that can be applied to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0108] like Figure 7As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An I / O interface 705 is also connected to the bus 704.

[0109] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to electronic device 700. Input unit 706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 707 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, disk and optical disk. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, WiFi devices, worldwide interoperability for microwave access (WiMax) devices, cellular communication devices, and / or the like.

[0110] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the data query method of the Spark engine described above can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. In some embodiments, the computing unit 701 can be configured to perform the data query method of the Spark engine described above by any other suitable means (e.g., by means of firmware).

[0111] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0112] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM) or flash memory, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0113] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device, PLD) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0115] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0116] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0117] Furthermore, it should be understood that the above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of this invention are still within the scope of this application.

Claims

1. A data query method using the Spark engine, characterized in that, include: In response to a data query request initiated by the target object, a task session is created for the target object; After confirming that the task session has been created, the data query request is parsed to determine the data query mode and data volume of the data to be queried. The initial configuration strategy is adjusted based on the data query pattern and the data volume to obtain the target configuration strategy; wherein, the initial configuration strategy includes: initial component configurations set for each component included in the Spark engine; The target configuration strategy is sent to the Spark engine through the task session, so that the Spark engine can query the data to be queried from the fixed storage space where the data to be queried is located based on the target configuration strategy.

2. The method as described in claim 1, characterized in that, The step of parsing the data query request to determine the data query pattern and data volume of the data to be queried includes: Send the data query request to the Hive engine; Receive Hive metadata returned by the Hive engine based on the data query request; the Hive metadata includes at least: the data query mode and the data volume of the data to be queried.

3. The method as described in claim 1, characterized in that, The adjustment of the initial configuration strategy based on the data query pattern and the data volume to obtain the target configuration strategy includes: The queue length and resource expansion information of the task queue are determined based on the amount of data; wherein, the task queue is used to store multiple data query subtasks corresponding to the data query request; Based on the queue length, the resource expansion information, and at least one task execution rule associated with the data query mode, the initial configuration strategy is adjusted to obtain the target configuration strategy.

4. The method as described in claim 3, characterized in that, The process of determining the queue length and resource expansion information of the task queue based on the data volume includes: The queue length of the task queue is determined based on the quantity range to which the data volume belongs, and the sub-data volume corresponding to each key included in the data to be queried is determined from the data volume. For each of the keys, perform the following operations: If the first sub-data volume corresponding to the first key is greater than a preset first data volume threshold, then based on the quantitative relationship between the first sub-data volume and the first data volume threshold, the first key is split into multiple sub-keys; wherein, the first key is any one of the keys. Based on the amount of sub-data corresponding to each of the multiple sub-keys, partitions are set for each of the multiple sub-keys; If the first sub-data volume is less than a preset second data volume threshold, then the partition set for the first key and the partition set for at least one second key included in each key will be merged; wherein, the sub-data volume corresponding to each second key is less than the second data volume threshold, and the second data volume threshold is less than the first data volume threshold.

5. The method as described in claim 1, characterized in that, The method further includes: Record the performance indicator parameter set corresponding to the multi-level performance indicators during the process of the Spark engine querying the data to be queried from the fixed storage space based on the target configuration strategy; Based on the mean value and threshold value of the performance index parameters corresponding to multiple performance index parameter sets, the performance index evaluation value corresponding to the multi-level performance index is determined. Based on the performance evaluation values ​​and weights corresponding to the multi-level performance indicators, the query performance evaluation value of the Spark engine is obtained.

6. The method according to any one of claims 1-5, characterized in that, After sending the target configuration strategy to the Spark engine through the task session, so that the Spark engine can query the data to be queried from the fixed storage space where the data to be queried is located based on the target configuration strategy, the method further includes: In response to the target object's data retrieval request for the data to be queried, based on at least one data element identifier carried in the data retrieval request, the data elements corresponding to the at least one data element identifier are retrieved from the data to be queried. At least one data element is displayed to the target object through the task session.

7. The method according to any one of claims 1-5, characterized in that, After sending the target configuration strategy to the Spark engine through the task session, so that the Spark engine can retrieve the data to be queried from the fixed storage space where the data to be queried is located based on the target configuration strategy, the method further includes: If the number of queries for the data to be queried within a set time range exceeds a set threshold, then a temporary cache space for the data to be queried is constructed based on the business scenario corresponding to the data to be queried. The data to be queried is compressed according to the set data format, and the compressed data to be queried is saved in the temporary cache space.

8. The method as described in claim 1, characterized in that, The components include at least one or a combination of: Cost Optimizer (CBO), Adaptive Query Execution (AQE), Dynamic Partition Pruning (DPP), and Shuffle.

9. A data query device for the Spark engine, characterized in that, include: The session creation module is used to create a task session for the target object in response to a data query request initiated by the target object; The request parsing module is used to parse the data query request after determining that the task session has been created, and to determine the data query mode and data volume of the data to be queried. The strategy adjustment module is used to adjust the initial configuration strategy based on the data query mode and the data volume to obtain the target configuration strategy; wherein, the initial configuration strategy includes: initial component configurations set for each component included in the Spark engine; The data query module is used to send the target configuration strategy to the Spark engine through the task session, so that the Spark engine can query the data to be queried from the fixed storage space where the data to be queried is located based on the target configuration strategy.

10. An electronic device, characterized in that, include: processor; A memory storing a program; wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1-8.

Images