Data query method and apparatus
By determining the user's queue in the target computing engine and allocating query requests in Kyuubi, the problem of inflexible user and resource management methods is solved, queue-level resource isolation and access control are achieved, and the system's management level and security are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI BILIBILI TECH CO LTD
- Filing Date
- 2022-10-19
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, Kyuubi's user and resource management methods are not very flexible and cannot effectively achieve queue-level resource isolation and access control.
By responding to user query requests, the system determines the user's target queue in the target computing engine, submits the query request to that queue for querying, obtains the query results, and uses Yarn for resource management to achieve queue-level resource isolation and access control.
It improves the flexibility of user and resource management, achieves queue-level resource isolation and access control, and enhances system security and management hierarchy.
Smart Images
Figure CN115640314B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data technology, and in particular to a data query method, apparatus, computer equipment, and storage medium. Background Technology
[0002] Kyuubi is an open-source project contributed to the Apache community by the NetEase team. It is mainly used in big data scenarios, including offline big data computing, ad hoc queries, and BI (business intelligence). It is a distributed, multi-user-supporting big data processing service that is compatible with JDBC (Java Database Connectivity) or ODBC (Open Database Connectivity).
[0003] Currently, before using Kyubi, new users need to be configured into a group through underlying configuration. When users use Kyubi to make queries, Kyubi then allocates resources and controls permissions according to the user's group. This method of user and resource management is relatively inflexible. Summary of the Invention
[0004] The purpose of this application is to provide a data query method, apparatus, computer device, and storage medium to solve the technical problem of low flexibility in user and resource management when using Kyubi.
[0005] One aspect of this application provides a data query method, comprising: in response to a user's query request, determining the target queue in a target computing engine, the target computing engine including a plurality of queues; submitting the query request to the target queue for querying; and obtaining the query results returned by the target queue.
[0006] Optionally, submitting the query request to the target queue for querying includes: allocating the query request to a target engine instance corresponding to the target queue; and submitting the query request to the target queue for querying through the target engine instance.
[0007] Optionally, the query request includes the user's ID, and submitting the query request to the target queue for querying includes: submitting the query request to the target queue based on the user's ID, so that the target queue can perform a query based on the user's ID.
[0008] Optionally, submitting the query request to the target queue based on the user's ID, so that the target queue can query based on the user's ID, includes: submitting the query request to the target queue, and binding the target thread in the target queue with the user's ID, so that the target thread can process the query request based on the user's ID.
[0009] Optionally, the method further includes: determining resource requirement parameters of the target queue; requesting resources from the target computing engine based on the resource requirement parameters to create the target queue in the target computing engine.
[0010] Optionally, the method further includes: listening to the target event of the query request; determining the status corresponding to the query request based on the target event; and displaying the status on a preset page in real time.
[0011] Optionally, the method further includes: collecting target information of the query request, the target information including user, session, job and stage information; determining progress and resource consumption information corresponding to the query request based on the target information; and sending the progress and resource consumption information back to the front end for display.
[0012] Optionally, the method is applied to Kyuubi, where the target computing engine is Spark.
[0013] Optionally, the method further includes: monitoring the memory usage of the target computing engine; storing the results pulled from the target computing engine to a local disk when the memory usage exceeds a preset threshold; and retrieving the pulled results from the local disk in batches and returning them in response to a request to obtain the results.
[0014] One aspect of this application provides a data query apparatus, comprising: a determining module, configured to determine, in response to a user's query request, the target queue in a target computing engine, the target computing engine including a plurality of queues; a query module, configured to submit the query request to the target queue for querying; and an obtaining module, configured to obtain the query results returned by the target queue.
[0015] One aspect of this application provides a computer device, the computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the data query method described above.
[0016] Another aspect of this application provides a computer-readable storage medium storing a computer program that can be executed by at least one processor to perform the steps of the data query method described above.
[0017] The data query method, apparatus, computer equipment, and storage medium provided in this application have the following advantages:
[0018] By responding to user query requests, the system determines the target queue in the target computing engine where the user resides, submits the query request to the target queue for processing, and then retrieves the query results returned by the target queue. Since query requests are processed through queues within the target computing engine, user permissions and resources can be managed on Yarn. This provides a higher level of management compared to Kyubi's original group mode, which relied on underlying configuration, thereby improving the flexibility of user and resource management. Furthermore, because query requests are only processed in the target queue corresponding to the user and do not affect data in other queues, queue-level resource isolation and access control can be achieved. Attached Figure Description
[0019] Figure 1 An environmental architecture diagram illustrating an embodiment of this application is shown schematically;
[0020] Figure 2 This is an example diagram illustrating the architecture of the data query method according to an embodiment of this application;
[0021] Figure 3 A flowchart illustrating the data query method of Embodiment 1 of this application is shown schematically;
[0022] Figure 4 for Figure 3 Flowchart of the sub-steps of step S520;
[0023] Figure 5 A diagram illustrating the principle of allocating query requests to a queue;
[0024] Figure 6 A diagram illustrating the principle of processing query requests as a user.
[0025] Figure 7 for Figure 3 Flowchart of the new step;
[0026] Figure 8 Example diagram for allocating queue resources;
[0027] Figure 9 for Figure 3 Flowchart of another new step;
[0028] Figure 10 for Figure 3 A flowchart for yet another new step;
[0029] Figure 11 for Figure 3 A flowchart for yet another new step;
[0030] Figure 12 A block diagram of the data query device according to Embodiment 2 of this application is shown schematically;
[0031] Figure 13 The schematic diagram illustrates the hardware architecture of the computer device according to Embodiment 3 of this application. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0033] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.
[0034] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.
[0035] The following is an explanation of the terms used in this application:
[0036] Kyubi is a high-performance, general-purpose JDBC (Java Database Connectivity) and SQL (Structured Query Language) execution engine. It provides a standardized JDBC interface for convenient data access in big data scenarios. End users can focus on developing business systems and mining data value without needing to understand the underlying big data platform (computing engine, storage service, metadata management, etc.). Kyubi relies on Apache Spark to provide high-performance data query capabilities, improves ad-hoc (ad hoc query) response capabilities through engine caching, and enhances concurrency through horizontal scaling and load balancing.
[0037] Spark is an open-source cluster computing framework. Unlike Hadoop MapReduce, which stores intermediate data to disk after completing its work, Spark uses in-memory computing technology, enabling analysis and computation within memory before data is written to disk. Spark's instruction cycle for executing programs in memory can be up to 100 times faster than Hadoop MapReduce, and even when executing programs to disk, Spark can still be up to 10 times faster.
[0038] Yarn (Yet Another Resource Negotiator) is a new Hadoop resource manager. It is a general-purpose resource management system that provides unified resource management and scheduling for upper-layer applications. Its introduction has brought great benefits to the cluster in terms of utilization, unified resource management and data sharing.
[0039] Kubernetes (often abbreviated as K8s) is an open-source system for automatically deploying, scaling, and managing containerized applications.
[0040] HDFS is the Hadoop Distributed File System, which enables reliable distributed reading and writing of large-scale data.
[0041] Flink is an open-source stream processing framework developed by the Apache Software Foundation. Its core is a distributed streaming data stream engine written in Java and Scala, which executes arbitrary streaming data programs in a data parallel and pipeline manner. Its pipeline runtime system can execute batch processing and stream processing programs.
[0042] Hive is a data warehouse built on the Hadoop architecture that provides data refinement, querying, and analysis.
[0043] Ad-hoc (ad-hoc query) allows users to flexibly select query conditions according to their needs, and the system can generate corresponding statistical reports based on the user's selection. The biggest difference between ad-hoc and ordinary application queries is that ordinary application queries are custom-developed, while ad-hoc queries are based on user-defined query conditions.
[0044] Figure 1 An environmental architecture diagram illustrating an embodiment of this application is shown in the figure:
[0045] The user submits a query request through client 100. After receiving the query request from client 100, query engine 200 determines the target queue in computing engine 300 where the user is located; submits the query request to the target queue in computing engine 300; computing engine 300 queries data in database 400 through the target queue and returns the query results to query engine 200, thereby enabling query engine 200 to obtain the corresponding query results; after obtaining the corresponding query results, query engine 200 returns the query results to client 100.
[0046] In an exemplary embodiment, client 100 may include devices such as mobile devices, tablets, laptops, smart devices (e.g., smart clothing, smartwatches, smart glasses), virtual reality headsets, gaming devices, set-top boxes, digital streaming devices, robots, in-vehicle terminals, smart TVs, TV boxes, or e-book readers.
[0047] The query engine 200 includes, but is not limited to, query execution engines such as kyuubi; the computing engine 300 includes, but is not limited to, Flink, Spark, or Hive; and the database 400 includes, but is not limited to, HDFS.
[0048] Please refer to Figure 2 This is an example architecture diagram of the data query method in this application. As shown in the diagram, the user's query request is submitted to Kyubi through the Dispatcher. After Kyubi determines the queue in Spark, it submits the query request to the target queue in Spark for querying. The target queue in Spark queries HDFS according to the query request to obtain the corresponding query results and return them to Kyubi.
[0049] In related technologies, before using Kyubi, new users need to be configured into a group through underlying configuration. When users use Kyubi, Kyubi allocates resources and controls permissions according to the group to which the user belongs, resulting in low flexibility in user and resource management.
[0050] The data query method of this application embodiment can improve the flexibility of user and resource management when using Kyubi.
[0051] The data query scheme of this application will be described below through several embodiments. For ease of understanding, the following will use examples... Figure 1 The query engine 200 in the example is described as the execution subject.
[0052] Example 1
[0053] Figure 3 The flowchart of the data query method of Embodiment 1 of this application is illustrated, including steps S510 to S530, which are described in detail below:
[0054] Step S510: In response to the user's query request, determine the target queue in the target computing engine where the user is located. The target computing engine includes several queues.
[0055] Optionally, when each user registers, each user is associated with a target computing engine (corresponding to...). Figure 1 The query engine 200 corresponds to the queue in the computing engine 300, so that after receiving a user's query request, it can determine the target queue in the target computing engine based on the user's registration information. The query request may include relevant user information, so that the query engine 200 can determine the target queue in the target computing engine based on the user's relevant information.
[0056] In practical applications, the correspondence between a user's department and a queue in the target computing engine can be determined based on information such as the user's department. For example, department A corresponds to queue A, department B corresponds to queue B, and if a user belongs to department B, then the user is matched with queue B.
[0057] The target computing engine can create a corresponding number of queues based on the actual application environment, such as creating queues according to different departments or different businesses, so that the created queues can be adapted to the needs of departments or businesses.
[0058] Step S520: Submit the query request to the target queue for querying.
[0059] After determining the target queue in which the user belongs, the query engine 200 can submit the query request to the target queue for querying. For example, if the user corresponds to queue A in the target computing engine, then queue A is the target queue for the user. The query engine 200 submits the query request to queue A in the target computing engine, and queue A processes the query request to achieve the corresponding data query.
[0060] Step S530: Obtain the query results returned by the target queue.
[0061] The target queue queries the corresponding database based on the query request, and then returns the query results to the query engine 200, so that the query engine 200 can obtain the query results returned by the target queue.
[0062] The data query method in this application responds to a user's query request, determines the target queue in the target computing engine where the user is located, submits the query request to the target queue for querying, and then obtains the query results returned by the target queue. Since the query request is queried through a queue in the target computing engine, user permissions and resources can be managed on Yarn. Therefore, compared with Kyubi's original group mode configured at the underlying level, it has a higher management level, thereby improving the flexibility of user and resource management. At the same time, since the query request is only assigned to the target queue corresponding to the user for processing and does not affect the data in other queues, queue-level resource isolation and access control can be achieved.
[0063] In an exemplary embodiment, the data query method is applied to Kyubi, and the target computing engine is Spark. By applying the data query method in Kyubi, queue-level resource isolation and access control can be achieved in a system composed of Kyubi and Spark.
[0064] In an exemplary embodiment, such as Figure 4 As shown, step S520 may include steps S521 to S522, as detailed below:
[0065] Step S521: Assign the query request to the target engine instance corresponding to the target queue.
[0066] Optionally, the query engine 200 has several engine instances. One queue in the target computing engine can correspond to one or more engine instances. The engine instance establishes a connection with the corresponding queue through a session. When the query engine 200 distributes query requests to the target engine instance corresponding to the target queue, if there is only one target engine instance, the query request can be directly distributed to the target engine instance. If there are two or more target engine instances, the distribution can be random or based on the load of the target engine instances. For example, query requests can be preferentially distributed to target engine instances with lighter loads. The specific settings can be configured according to actual needs and are not limited here.
[0067] In practical applications, the query engine 200 may also include several server instances. When a query request is received from the upstream dispatcher, the query request may be randomly or distributed to a server instance based on the load, and then distributed to the target engine instance corresponding to the target queue.
[0068] Step S522: Submit the query request to the target queue for querying through the target engine instance.
[0069] Since the target engine instance establishes a connection with the target queue through a session, after receiving a query request, the target engine instance can directly submit the query request to the target queue for querying and obtain the corresponding query results.
[0070] Please refer to Figure 5 This diagram illustrates the principle of allocating query requests to queues. As shown, after receiving a query request from the upstream Dispatcher, Kyubi first randomly assigns the request to a Kyubi server instance (KyuubiServerInstance) on the ServerDiscoverySide. Then, on the EngineDiscoverySide, it allocates the request to the corresponding engine instance (KyuubiEngineInstance). Finally, Kyubi uses its session with the Spark computing engine to distribute the request to the appropriate queue. In the diagram, engine instances are registered in the ZooKeeper namespace, corresponding to the game, live, and report queues in Spark. Upon receiving a query request, Kyubi allocates it to the appropriate engine instance based on the mapping between the query request and the engine instance.
[0071] In this embodiment, by allocating the query request to the target engine instance corresponding to the target queue, and then submitting the query request to the target queue through the target engine instance, the query request can be allocated to the corresponding queue, thereby effectively achieving queue-level resource isolation.
[0072] In an exemplary embodiment, the query request includes the user's ID. In step S520, submitting the query request to the target queue for querying may include: submitting the query request to the target queue based on the user's ID, so that the target queue can perform a query based on the user's ID.
[0073] It should be understood that currently, the Kyubi server is generally started as a superuser (e.g., Hive). In Spark scenarios, the driver and executors share the same username. Different users submit different SQL queries, and neither the driver nor the executor can distinguish which user submitted the current task. This poses security risks regarding data, resource allocation, and access control. Therefore, including the user's ID in the query request and submitting the query to the target queue based on the user's ID allows for differentiation of user operations, effectively managing user permissions and improving security in areas such as data, resource allocation, and access control.
[0074] In an exemplary embodiment, the above-mentioned submission of the query request to the target queue based on the user's ID, so that the target queue can perform the query based on the user's ID, may include: submitting the query request to the target queue and binding the target thread in the target queue with the user's ID, so that the target thread can process the query request based on the user's ID.
[0075] Taking Kyuubi as the query engine and Spark as the target computing engine as an example, the specific details could be:
[0076] 1. Upstream of Kyubi, the Dispatcher can submit query requests (SQL) using the usernameproxyUser identity;
[0077] 2. On the KyuubiServer side, KyuubiServer can be started using the Hiveprincipal identity;
[0078] 3. On the Spark Engine side, the Driver and Executor can be started using the Hive identity. The Driver submits SQL as the usernameproxyUser, and the Executor starts Task threads to execute Tasks as the usernameproxyUser. Simultaneously, ensure that the UGI (usergroup information) bound to all public thread pools is correct. Please refer to [link / reference]. Figure 6 This is an example diagram illustrating the principle of handling query requests with a user identity. As shown in the figure, the corresponding submissions are made in the ExecuteStatement of SparkEngine(Driver) and the Executor of SparkExecutor, respectively, using usernameproxyUser.
[0079] In other words, the user identity corresponding to the query request is obtained from the upstream, and the corresponding thread is bound with the corresponding user identity, thereby achieving the purpose of distinguishing different users and implementing different access control for different users.
[0080] In this embodiment, by submitting query requests to the target queue and binding the target thread in the target queue with the user's ID, the target thread can process the query requests according to the user's identity. This allows for the differentiation and binding of user identities for query requests from different users, thereby effectively managing the permissions of different users and improving data security.
[0081] In an exemplary embodiment, such as Figure 7 As shown, the data query method may also include steps S540 to S550, as detailed below:
[0082] Step S540: Determine the resource requirement parameters of the target queue.
[0083] The resource requirements of queues can be managed through a configuration center, allowing different queues to be configured with different resource requirements as needed. For example, more resources can be allocated to queues with a large workload, while fewer resources can be allocated to queues with a small workload. Resource requirements can include parameters related to storage resources and computing resources (such as CPU).
[0084] Step S550: Request resources from the target computing engine based on resource requirement parameters to create a target queue in the target computing engine.
[0085] In practical applications, when the queue starts the corresponding engine for the first time, the resource requirement parameters can be appended to the startup command to overwrite the parameters, and then a queue with the corresponding resources can be created based on the resource requirement parameters.
[0086] Please refer to Figure 8 This diagram illustrates the allocation of queue resources. As shown, the queue for "game" receives a relatively large amount of resources due to its large workload, with resource requirements of sparkdriver.memory = 100g and spark executor.memory = 16g. Conversely, the queue for "report" receives a relatively small amount of resources due to its smaller workload, with resource requirements of sparkdriver.memory = 50g and spark executor.memory = 8g. Based on these resource requirements, the corresponding queue engines are then created on Yarn or Kubernetes to obtain the appropriate queues.
[0087] In this embodiment, by determining the resource requirement parameters of the target queue, resources of the target computing engine are requested according to the resource requirement parameters to create the target queue in the target computing engine. Different queues can be created in the target computing engine according to the pre-set resource requirement parameters, so that the resources corresponding to the created queues meet the requirements of the corresponding tasks.
[0088] In an exemplary embodiment, such as Figure 9 As shown, the data query method may also include steps S610 to S630, as detailed below:
[0089] Step S610: Listen for the target event of the query request.
[0090] Since the query request status needs to be determined based on events, the target event can be a status-related event, such as Spark's Job, Stage, and Task events, as well as Spark's SQL-related events, such as SessionCreate, SessionClose, executionStart, executionRunning, and executionEnd. The target event is related to the status that needs to be determined; therefore, the appropriate target event can be determined based on the actual required status, without specific restrictions.
[0091] When listening to the target event of a query request, you can use a custom listener (such as Kyuubilistener) to listen for the corresponding event.
[0092] Step S620: Determine the status corresponding to the query request based on the target event.
[0093] Based on monitoring target events, the status corresponding to the query request can be obtained directly from the target events. Alternatively, the monitored target events can be statistically analyzed and stored, and the status corresponding to the query request can be determined based on the statistical and analytical results.
[0094] Step S630: Display the status in real time on a preset page.
[0095] The preset page can include information related to the status of the query request, such as the user's ID, IP address, session ID, start time, end time, and duration. The specific settings can be configured according to actual needs, and no specific restrictions are imposed here.
[0096] In this embodiment, by listening to the target event of the query request, the status corresponding to the query request is determined based on the target event, and the status is displayed in real time on a preset page. This allows users to understand the real-time status of the query request on the preset page, improving the user-friendliness of human-computer interaction. In addition, user information can also be displayed on the preset page as needed, making it easier to distinguish query requests submitted by different users.
[0097] In an exemplary embodiment, such as Figure 10 As shown, the data query method may also include steps S640 to S660, as detailed below:
[0098] Step S640: Collect the target information of the query request. The target information includes user, session, job and stage information.
[0099] When collecting target information for a query request, it can be done based on events. For example, the target information for a query request can be collected by listening to events as described above.
[0100] In addition to user, session, job, and stage information, target information can also include other information. The specific settings can be configured according to actual needs, and there are no restrictions here.
[0101] Step S650: Determine the progress and resource consumption information corresponding to the query request based on the target information.
[0102] Optionally, based on the collected target information, a scheduled task can be used to correlate the collected information and calculate resource consumption to determine the progress and resource consumption information corresponding to the query request. The resource consumption information may include memory usage, CPU usage, and the amount of data read.
[0103] Step S660: The progress and resource consumption information are sent back to the front end for display.
[0104] For example, progress and resource consumption information can be injected into the operation log and sent back to the front-end log for display.
[0105] In this embodiment, by collecting the target information of the query request, determining the progress and resource consumption information corresponding to the query request based on the target information, and then sending the progress and resource consumption information back to the front end for display, users can promptly understand the progress and resource consumption information corresponding to the query request, and then make an effective assessment of the health status of the task based on the progress and resource consumption information.
[0106] In an exemplary embodiment, such as Figure 11As shown, the data query method may also include steps S710 to S730, as detailed below:
[0107] Step S710: Monitor the memory usage of the target computing engine.
[0108] In step S720, if the memory usage exceeds a preset threshold, the results pulled from the target computing engine will be stored on the local disk.
[0109] In step S730, in response to the request to obtain results, the results are retrieved in batches from the local disk and returned.
[0110] In this embodiment, by monitoring the memory usage of the target computing engine, when the memory usage exceeds a preset threshold, the results pulled from the target computing engine are stored on the local disk. When the user needs to obtain the results, the results are retrieved from the local disk in batches according to the user's request and returned. This avoids the problem of excessive memory usage of the pulled results causing a decrease in the performance of the target computing engine and affecting the user's query experience, thereby improving the stability of the system.
[0111] Example 2
[0112] Figure 12 The diagram illustrates a block diagram of a data query device 800 according to Embodiment 2 of this application. This data query device 800 can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program modules referred to in the embodiments of this application are a series of computer program instruction segments capable of performing specific functions. The following description will specifically introduce the functions of each program module in this embodiment.
[0113] like Figure 12 As shown, the data query device 800 may include a determining module 810, a querying module 820, and an obtaining module 830.
[0114] The determination module 810 is used to determine the target queue in the target computing engine in response to a user's query request, wherein the target computing engine includes several queues;
[0115] Query module 820 is used to submit the query request to the target queue for querying;
[0116] The acquisition module 830 is used to acquire the query results returned by the target queue.
[0117] In an exemplary embodiment, the query module 820 is further configured to: allocate the query request to a target engine instance corresponding to the target queue; and submit the query request to the target queue for querying through the target engine instance.
[0118] In an exemplary embodiment, the query request includes a user's ID, and the query module 820 is further configured to: submit the query request to the target queue based on the user's ID, so that the target queue can perform a query based on the user's ID.
[0119] In an exemplary embodiment, the query module 820 is further configured to: submit the query request to the target queue, and bind the target thread in the target queue to the user's ID, so that the target thread can process the query request according to the user's identity.
[0120] In an exemplary embodiment, the data query device 800 further includes a request module (not shown in the figure), wherein the request module is configured to: determine the resource requirement parameters of the target queue; and request resources of the target computing engine according to the resource requirement parameters to create the target queue in the target computing engine.
[0121] In an exemplary embodiment, the data query device 800 further includes a first display module (not shown in the figure), wherein the first display module is used to: listen to the target event of the query request; determine the status corresponding to the query request based on the target event; and display the status on a preset page in real time.
[0122] In an exemplary embodiment, the data query device 800 further includes a second display module (not shown in the figure), wherein the second display module is used to: collect target information of the query request, the target information including user, session, job and stage information; determine the progress and resource consumption information corresponding to the query request based on the target information; and send the progress and resource consumption information back to the front end for display.
[0123] In an exemplary embodiment, the data query device 800 is applied to Kyuubi, and the target computing engine is Spark.
[0124] In an exemplary embodiment, the data query device 800 further includes a transfer module (not shown in the figure), wherein the transfer module is used to: monitor the memory usage of the target computing engine; if the memory usage exceeds a preset threshold, store the results pulled from the target computing engine to a local disk; and in response to a request to obtain results, retrieve the pulled results from the local disk in batches and return them.
[0125] Example 3
[0126] Figure 13 The diagram schematically illustrates the hardware architecture of a computer device 900 suitable for a data query method according to Embodiment 3 of this application. The computer device 900 can be a device capable of automatically performing numerical calculations and / or data processing according to pre-set or stored instructions. For example, it can be a rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), gateway, etc. Figure 13 As shown, the computer device 900 includes, but is not limited to, a memory 910, a processor 920, and a network interface 930 that can communicate with each other via a system bus. Wherein:
[0127] The memory 910 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 910 may be an internal storage module of the computer device 900, such as the hard disk or memory of the computer device 900. In other embodiments, the memory 910 may also be an external storage device of the computer device 900, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 910 may include both the internal storage module and the external storage device of the computer device 900. In this embodiment, the memory 910 is typically used to store the operating system and various application software installed on the computer device 900, such as program code for data query methods. In addition, the memory 910 can also be used to temporarily store various types of data that have been output or will be output.
[0128] In some embodiments, processor 920 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 920 is typically used to control the overall operation of computer device 900, such as performing control and processing related to data interaction or communication with computer device 900. In this embodiment, processor 920 is used to run program code stored in memory 910 or process data.
[0129] Network interface 930 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 900 and other computer devices. For example, network interface 930 is used to connect computer device 900 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 900 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.
[0130] It should be pointed out that, Figure 13 Only computer devices with components 910-930 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.
[0131] In this embodiment, the data query method stored in the memory 910 can be further divided into one or more program modules and executed by one or more processors (processor 920 in this embodiment) to complete the embodiment of this application.
[0132] Example 4
[0133] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data query method in the embodiments.
[0134] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the data query method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.
[0135] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.
[0136] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A data query method, characterized in that, include: In response to a user's query request, determine the target queue in the target computing engine where the user is located; the target computing engine includes several queues. Submit the query request to the target queue for querying; Obtain the query results returned by the target queue; The query request includes the user's ID, and submitting the query request to the target queue for querying includes: The query request is submitted to the target queue based on the user's ID, so that the target queue can perform a query based on the user's ID; The step of submitting the query request to the target queue based on the user's ID, so that the target queue can perform a query based on the user's ID, includes: The query request is submitted to the target queue, and the target thread in the target queue is bound to the user's ID so that the target thread can process the query request according to the user's identity.
2. The data query method according to claim 1, characterized in that, Submitting the query request to the target queue for querying includes: The query request is assigned to the target engine instance corresponding to the target queue; The query request is submitted to the target queue for querying through the target engine instance.
3. The data query method according to claim 1, characterized in that, Also includes: Determine the resource requirement parameters of the target queue; The resource requirement parameters are used to request resources from the target computing engine to create the target queue within the target computing engine.
4. The data query method according to claim 1, characterized in that, Also includes: Listen for the target event of the query request; Determine the status corresponding to the query request based on the target event; The status is displayed in real time on a preset page.
5. The data query method according to claim 4, characterized in that, Also includes: Collect the target information of the query request, including user, session, job, and stage information; Based on the target information, determine the progress and resource consumption information corresponding to the query request; The progress and resource consumption information are sent back to the front end for display.
6. The data query method according to any one of claims 1-5, characterized in that, Applied to Kyuubi, the target computing engine is Spark.
7. The data query method according to any one of claims 1-5, characterized in that, Also includes: Monitor the memory usage of the target computing engine; If the memory usage exceeds a preset threshold, the results retrieved from the target computing engine will be stored on the local disk. In response to a request to obtain results, the retrieved results are obtained in batches from the local disk and returned.
8. A data query device, characterized in that, include: The determination module is used to determine the target queue in the target computing engine in response to a user's query request, wherein the target computing engine includes several queues; The query module is used to submit the query request to the target queue for querying; The acquisition module is used to acquire the query results returned by the target queue; The query request includes the user's ID, and submitting the query request to the target queue for querying includes: The query request is submitted to the target queue based on the user's ID, so that the target queue can perform a query based on the user's ID; The step of submitting the query request to the target queue based on the user's ID, so that the target queue can perform a query based on the user's ID, includes: The query request is submitted to the target queue, and the target thread in the target queue is bound to the user's ID so that the target thread can process the query request according to the user's identity.
9. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data query method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the data query method according to any one of claims 1 to 7.
11. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1 to 7.