Task processing method, electronic device and storage medium
By storing engine registration information on Hive's server side, reusing or creating a computing engine to process client's task requests, the problem of recreating the computing engine for each session in Hive is solved, and task processing efficiency and response speed are improved.
Patent Information
- Application Number
- CN202411049489.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-01
AI Technical Summary
In Hive's data query method, every time a user initiates a session, the computing engine needs to be recreated, resulting in low task processing efficiency, especially in improvised query scenarios, it is difficult to meet the needs of efficient operation.
By storing engine registration information on the server side, obtaining the history engine information corresponding to the client information. If it exists, reuse the historical computing engine to process the task request; if it does not exist, create a new computing engine and store its corresponding relationship.
It realizes the multiplexing of the computing engine, saves the startup time of the computing engine, improves task processing efficiency, and significantly improves the response speed in improvised query scenarios.
Smart Images

Figure CN118567821B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of big data processing, and in particular to a task processing method, electronic device and storage medium. Background Art
[0002] Hive is a data warehouse tool based on Hadoop that can store, query and analyze large-scale data stored in Hadoop. For example, it converts SQL statements into a series of tasks that can be executed in the Hadoop cluster, thereby realizing functions such as data query.
[0003] At present, in the process of Hive-based data query method, each time it receives a request from a user, HiveServer2 will create a session for the request, and create a set of computing engines for the session through the cluster resource manager (such as Yarn, K8s, etc.), and the computing engine will execute the corresponding task. However, after the task is completed, the user will close the session with HiveServer2, and the corresponding computing engine will also be closed to release computing resources; if the user initiates a session again, the computing engine needs to be re-created and wait for a period of time for the computing engine to start, which greatly affects the efficiency of task processing.
[0004] Especially in the scenario of ad hoc query, users usually need to conduct interactive query and analysis quickly, that is, they need to start the query engine quickly. However, the current Hive architecture is difficult to meet the requirements of efficient operation under big data. Summary of the invention
[0005] The present application at least provides a task processing method, apparatus, device and computer-readable storage medium.
[0006] The first aspect of the present application provides a task processing method, which is applied to a server side and includes: in response to a target task request received from a client, obtaining client information carried by the target task request; in the stored engine registration information, obtaining historical engine information corresponding to the client information; wherein the historical computing engine corresponding to the historical engine information has processed the task request of the client; in response to successfully obtaining the historical engine information, sending the target task request to the historical computing engine for task processing; in response to a failure to obtain the historical engine information, creating a new computing engine for processing the target task request, using the new computing engine to process the target task request, and storing the correspondence between the client information and the new computing engine in the engine registration information.
[0007] In one embodiment, the server receives the target task request through a target session, the historical engine information includes an engine network address and an engine port, and sending the target task request to the historical computing engine for task processing includes: acquiring a distributed lock corresponding to the client based on the target session; in response to successful acquisition of the distributed lock, determining a corresponding historical computing engine based on the engine network address and the engine port; and sending the target task request received through the target session to the historical computing engine for task processing.
[0008] In one embodiment, the server receives the target task request through a target session, and the creation of a new computing engine for processing the target task request includes: obtaining a distributed lock corresponding to the client based on the target session; in response to successful acquisition of the distributed lock, creating a new computing engine corresponding to the client information.
[0009] In one embodiment, after creating a new computing engine corresponding to the client information, the method further includes: starting each engine component in the new computing engine; and releasing the distributed lock of the client in response to each engine component starting successfully.
[0010] In one embodiment, starting each engine component in the newly created computing engine includes: starting a registration request receiver in the newly created computing engine; sending an engine registration request corresponding to the target session to the registration request receiver, so that the registration request receiver creates context resources for the target session; using the newly created computing engine to process the target task request includes: sending the target task request to the newly created computing engine, so that the newly created computing engine uses the context resources of the target session to process the target task request.
[0011] In one embodiment, after the response to the received target task request initiated by the client, the method further includes: establishing a target session with the client based on the target task request initiated by the client; and allocating a matching service component to the target session based on the running load of the server.
[0012] In one embodiment, the server side receives the target task request through a target session. After sending the target task request to the historical computing engine for task processing, the method further includes: in response to the disconnection between the target session and the historical computing engine, recording the disconnection duration; if the disconnection duration is greater than a first time threshold, performing resource release processing on the working node in the historical computing engine; if the disconnection duration is greater than a second time threshold, performing resource release processing on the historical computing engine; the second time threshold is greater than the first time threshold.
[0013] In one embodiment, after recording the disconnection duration in response to the disconnection between the target session and the historical computing engine, the method further includes: in response to the disconnection duration being less than or equal to the first time threshold and receiving a current task request initiated by the client, establishing a current session with the client; connecting the current session to the historical computing engine, and sending the current task request received based on the current session to the historical computing engine, so that the historical computing engine processes the current task request.
[0014] The second aspect of the present application provides a task processing device, including: a request receiving module, used to respond to a target task request received from a client and obtain the client information carried by the target task request; an engine information acquisition module, used to obtain historical engine information corresponding to the client information in the stored engine registration information; wherein the historical computing engine corresponding to the historical engine information has processed the task request of the client; a task processing module, used to send the target task request to the historical computing engine for task processing in response to successful acquisition of the historical engine information; an engine creation module, used to create a new computing engine for processing the target task request in response to a failure to obtain the historical engine information, use the new computing engine to process the target task request, and store the correspondence between the client information and the new computing engine in the engine registration information.
[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned task processing method.
[0016] A fourth aspect of the present application provides a computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, the above-mentioned task processing method is implemented.
[0017] The above scheme, through the service component set in the server side, responds to the target task request initiated by the client, obtains the client information carried by the target task request; obtains the historical engine information corresponding to the client information in the engine registration information stored in the storage system of the server side; wherein the historical computing engine corresponding to the historical engine information has processed the task request of the client; if the acquisition fails, it indicates that the historical computing engine corresponding to the client does not exist in the server, therefore, the service component is required to perform engine creation processing based on the target session to obtain a newly created computing engine to process the target task request sent by the target session; if the acquisition is successful, it indicates that the historical computing engine corresponding to the client exists in the server, then the service component connects the historical computing engine to the client through the target session, so that the historical computing engine can process the target task request sent by the client through the target session; thereby, the computing engine can be reused, the startup time of the computing engine is saved, and the task processing efficiency is improved.
[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.
[0020] Figure 1 It is a flowchart of a traditional application scenario in the task processing method of this application;
[0021] Figure 2 is a flowchart of an exemplary embodiment of the task processing method of the present application;
[0022] Figure 3 is an exemplary overall flow chart of the task processing method of the present application;
[0023] Figure 4 It is a schematic diagram of the main modules of an exemplary computing engine driver in the task processing method of the present application;
[0024] Figure 5 is a block diagram of a task processing device shown in an exemplary embodiment of the present application;
[0025] Figure 6 It is a structural schematic diagram of an embodiment of the electronic device of the present application;
[0026] Figure 7 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION
[0027] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.
[0028] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.
[0029] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.
[0030] For ease of understanding, one of the applicable scenarios of the task processing method of the present application is now exemplified. Hive on Something provides a SQL on Hadoop solution based on the Hive query engine. For each request from the user, HiveServer (a service component based on Hive, and HiveServer2 is mainly used as a service component as an example in the following text) will create a session for each request, and create a set of computing engines for the session through a cluster resource manager (such as Yarn, K8s, etc.) (i.e., the aforementioned Something, currently Hive supports Spark, MapReduce, Tez and other execution engines). Among them, the execution engine may include a driver node and an execution node. The driver node is responsible for receiving tasks and scheduling tasks to each execution node for processing. Usually, the user will close the session with HiveServer2 after the query task is completed. Accordingly, the corresponding computing engine will be closed and destroyed to release computing resources; if the user initiates a session again, a set of computing engines needs to be recreated, so each time the user reconnects, it is necessary to wait for a period of time for the computing engine to start, which greatly affects the task processing efficiency.
[0031] Especially in the scenario of ad hoc query, users usually need to perform interactive query and analysis quickly, so there is an urgent need for a query engine that can be started quickly. Obviously, the current Hive on Something architecture cannot meet the current ad hoc query scenarios under big data.
[0032] For example, the overall process of using the HiveSQL system in the existing method can be referred to as follows Figure 1 As shown, Figure 1 This is a flowchart of a traditional application scenario in the task processing method of this application. In this case, the same user usually creates several sessions with HiveServer2 (for example, user 0-session 0, user 0-session 1, user 0-session 2, etc.). When HiveServer2 processes each session, it will create a corresponding computing engine to process the SQL tasks subsequently submitted by the user. Each computing engine contains a set of computing resources (such as Figure 1 ). In the specific implementation of the existing method, for example, Session 0 of User 0 has created a computing engine (User 0-Session 0 driver). If the user disconnects the session after using the session for a period of time, the computing resources (computing engine) corresponding to "User 0-Session 0" will be immediately released and destroyed, and the resources will be recycled and cannot be used again. If User 0 needs to connect to the HiveServer2 cluster again and establish a Session 1 connection, HiveServer2 will reapply for computing resources to create a new computing engine, and the time to recreate the computing engine will be relatively long, which at least includes program file resource preparation time, container scheduling time, container creation time, driver startup time, worker node creation time, etc. User 0 needs to wait for the new computing engine to be ready before sending a task request to the server through the current session so that the computing engine in the server can execute the SQL task.
[0033] Based on the above problems, this application provides a task processing method with higher task processing efficiency. Figure 2 , Figure 2 1 is a flowchart of an exemplary embodiment of the task processing method of the present application. The task processing method of the present application is applied to the server side. Specifically, it may include the following steps:
[0034] Step S110 , in response to receiving a target task request initiated by a client, obtaining client information carried in the target task request.
[0035] Among them, the client may include but is not limited to terminal devices such as mobile phones, computers, smart watches, etc., and the client information may include but is not limited to customer name, customer ID, customer IP address and other customer-level information, which is not limited here.
[0036] A task request refers to a task request sent by a client to a server during task processing. It should be noted that the server may receive one or more task requests initiated by the same or different clients at the same time or at different times. For multiple task requests, at least a first task request and a second task request may be included, and a third task request and more may be included. The target task request may be one or more of the task requests, which will not be described in detail here. The timing of the first task request is earlier than the timing of the second task request. Exemplarily, in response to receiving a first task request from a client, a first session (a session between a client and a server) corresponding to the first task request may be created. In response to receiving a second task request from a client, a second session corresponding to the second task request may be created. Based on the target task request, a target session may be created accordingly, and the type of the target session of the present application may refer to the first session and / or the second session, etc. The definition of "first" is to distinguish it from the "second" of the second task request in the following text in terms of the timing relationship, and no other attributes are defined. Specifically, the first task request may be the first task request or the Nth task request (N is a positive integer) sent to the server during the entire task processing process corresponding to the client, which is not defined here. Among them, the client sends a task request to the server so that the server can perform task execution processing accordingly according to the received task request. For example, a user can send a SQL (Structured Query Language) query request to the server. After receiving the SQL query request, HiveServer2 on the server will convert the SQL into several computing (query) tasks (specific task forms may include but are not limited to Spark tasks, Tez tasks, MapReduce tasks, etc.), and then send the computing tasks to the computing engine corresponding to the user on the server for task execution.
[0037] Therefore, during the execution of this step, HiveServer2 on the server side responds to the target task request sent by the client and creates a corresponding target session based on the target task request, which is equivalent to establishing a communication connection between the client and the server. Figure 3 As shown, Figure 3 This is an exemplary overall flow chart of the task processing method of the present application. It mainly shows the overall process of the first session (session 0) and the second session (session 1). It should be noted that a task request can correspond to a session, and a client can initiate multiple task requests, so a client can correspond to one or more sessions. Figure 3As shown in the figure, user 0 can initiate multiple sessions to the HiveServer2 cluster on the server side, and then session 0, session 1, and session 2 can be obtained. Specifically, in the process of creating a session, the client will first connect to the load balancing gateway on the server side, and then the load balancing gateway will assign a suitable service address (HiveServer2 address) to the client for connection based on the load pressure or operating pressure of each service component in the HiveServer2 cluster. (The same HiveServer2 service address can also be assigned to different users), thereby obtaining a session connection between the client and HiveServer2 on the server side, as shown in the figure. Figure 3 User 0-session 0 in the example connects to service component 0 (that is, HiveServer2-0). Service component 0 can then be used to process task requests sent by user 0 through session 0.
[0038] Step S120, obtaining historical engine information corresponding to the client information from the stored engine registration information; wherein the historical computing engine corresponding to the historical engine information has processed the task request of the client.
[0039] In conjunction with the above steps, the computing engine of the present application is set as a reusable computing engine. Therefore, when a task request from a client is received, the corresponding historical engine information will be searched based on the client information, thereby determining whether a historical computing engine that can be used by the client already exists in the server. If a historical computing engine corresponding to the client already exists, its engine information can be successfully obtained; if a historical computing engine corresponding to the client does not exist, its engine information cannot be successfully obtained. A historical computing engine refers to a computing engine that has been created and processed the client's task request, and historical engine information refers to the engine information of the historical computing engine.
[0040] For example, we can continue to refer to Figure 3 After session 0 is assigned the address of service component-0 (HiveServer2-0), session 0 establishes a connection with HiveServer2-0. As a result, HiveServer2-0 can provide services to the client through session 0, which is equivalent to HiveServer2-0 being responsible for processing the task request sent by the client through session 0. Specifically, the process of HiveServer2-0 providing services at least includes using the corresponding computing engine to execute and process the task request (such as SQL request) in session 0, and returning the processing result to the client through session 0.
[0041] Therefore, in this step, HiveServer2-0 needs to obtain the engine information of the computing engine corresponding to the client (that is, the historical engine information of the historical computing engine) in order to further accurately obtain the corresponding computing engine based on the engine information to perform its tasks. If there is no computing engine corresponding to the client in the server, it is necessary to create a computing engine for the client. Among them, the computing engine will generate engine information or the client's registration information for the computing engine after the first successful creation. The engine information can include the client information of the client at the time of creation, such as the client name, client ID, client IP address, etc., and will also include the IP address of the computing engine, and the service port, etc., which will not be repeated here.
[0042] It should be noted that in the present application, the way to obtain engine information can be to obtain it from a preset storage system, which can be a storage system located outside the server or a storage system located inside the server, which is not limited here. For example, if HiveServer2-1 successfully obtains engine information from the storage system when processing session 1 (for example, obtains engine information that matches the client name of the current client), the computing engine corresponding to the engine information (including the user 0 driver and the corresponding working node) is set as the computing engine that can be used by the client's session 1, and HiveServer2-1 is responsible for sending the task request sent by the client through session 1 to the computing engine for task processing.
[0043] Step S130, in response to successfully obtaining the historical engine information, sending the target task request to the historical calculation engine for task processing.
[0044] In conjunction with the above-mentioned embodiment, if the historical engine information corresponding to the client is successfully obtained, it indicates that there is still a historical computing engine in the server that can be used by the client. For example, when the second task request of the client is received, there is a historical computing engine that has processed the first task request of the client. Figure 3 , based on the second task request, a second session (user 0-session 1) is created between the client and the server, that is, HiveServer2-1 will search for the corresponding historical engine information in the storage system based on the information of the client that initiated the second session.
[0045] It should be noted that when receiving the first task request from the client, HiveServer2 creates a computing engine for the client for the first time, so HiveServer2-0 at that time cannot obtain the engine information from the storage system. However, in this scenario, the computing engine corresponding to the client has been created and its engine information is also stored in the storage system. Therefore, under normal circumstances, HiveServer2-1 can successfully obtain the engine information corresponding to the client from the storage system based on the second session. Then, HiveServer2-1 finds the corresponding computing engine based on the engine information it obtains, and by registering the computing engine, the computing engine and HiveServer2-1 can establish a connection, which is equivalent to HiveServer2-1 connecting the computing engine to the client through the second session. As a result, HiveServer2-1 can send the second task request sent by the client through the second session to the computing engine for task processing. Among them, the registration process can be understood as HiveServer2 initiating a registration request to the corresponding computing engine so that the computing engine registers the relevant information of HiveServer2, so that HiveServer2 and the computing engine can be connected.
[0046] It should be noted that, in the present application, the successful establishment of a connection between the computing engine and the client through the second session does not necessarily mean that the computing engine and the first session of the client are disconnected. The same computing engine can maintain a connection relationship with the first session, the second session and / or more sessions at the same time, which is not limited here. Whether the session and the computing engine are disconnected may depend on whether the client has disconnected a session, or whether a session has been unresponsive for a long time, etc., which will not be elaborated here.
[0047] Step S140, in response to a failure in obtaining historical engine information, a new computing engine for processing the target task request is created, the target task request is processed using the new computing engine, and the correspondence between the client information and the new computing engine is stored in the engine registration information.
[0048] In conjunction with the above steps, if the engine information cannot be successfully obtained from the storage system, it indicates that the storage system may not contain the engine information corresponding to the client, which means that the current server does not contain the computing engine that the client can use, for example, the client initiates a task request to the server for the first time. Therefore, it is necessary to create a computing engine for the client. For example, when the first task request from the client is received, there is no historical computing engine in the server. When the client first establishes a session with HiveServer2, since it is the first startup, the storage system usually does not contain the engine information corresponding to the client, that is, HiveServer2 cannot successfully obtain the historical engine information corresponding to the client from the storage system. Therefore, it is necessary to create a new computing engine for the client, use the newly created computing engine to process the target task request, and store the engine information of the computing engine and the client information in the storage system accordingly, so that when the server receives the task request initiated by the client again, HiveServer2 can find the corresponding historical computing engine through the client information to continue task processing.
[0049] Specifically, when HiveServer2 obtains the engine information corresponding to the client from the storage system based on the first session, if the acquisition fails, HiveServer2 needs to perform engine creation processing based on the first session to obtain a reusable computing engine (including but not limited to Spark, Tez, MapReduce and other computing engines), that is, a new computing engine. Reusable means that if the client continues to initiate task requests through other sessions, the other sessions of the client can be connected to the computing engine to implement task execution processing for different sessions of the same client by the same computing engine. For details, please refer to the implementation process after successfully obtaining the engine information of the historical computing engine based on the second session in the aforementioned steps.
[0050] Further, combining the implementation process of the above steps, the second task request is equivalent to the task request received from the client after receiving the first task request from the client, the timing of the second task request is later than the first task request, and the first task request and the second task request are task requests initiated by the client to the server based on different sessions. Exemplarily, if the first task request is the Nth request (N≥1, and is a positive integer) initiated by the client to HiveServer2, then the subsequent task request can be the N+Mth request initiated by the client to HiveServer2, and M is a positive integer greater than or equal to 1. Therefore, in an example scenario, if the first task request is the first request initiated by the client to the server, the acquisition of engine information fails due to the absence of a historical computing engine, and a new computing engine needs to be created to process the first task request. When the second task request from the same client is received, since the computing engine has been created, the engine information can be successfully obtained, and the second task request can be processed based on the existing computing engine. It should be noted that the historical computing engine and the newly created computing engine of the present application can be the same computing engine or different computing engines, depending on whether the computing engine is changed, deregistered, etc. during the period from when the newly created computing engine is called to when the historical computing engine is called. Therefore, in addition to processing the target task request, the newly created computing engine and the historical computing engine can also be used to process other task requests, which is not limited here.
[0051] Similar to the above steps, if the server receives the second task request from the client, a second session will be established between the client and the server, for example Figure 3 User 0-session 1 in the table; then assign the HiveServer2 service address to session 1 to obtain HiveServer2-1.
[0052] In summary, in the method of the present application, by trying to obtain the engine information corresponding to the client, it is determined whether there is a historical computing engine in the server that can be reused by the client; if so, the historical computing engine is used to process the client's target task request. This avoids the traditional method in which the computing engine is deregistered and recycled after processing the task request of the first session, resulting in the need to recreate the computing engine or start the computing engine when processing the second task request of the second session, thereby wasting more waiting time and reducing task processing efficiency. Especially in the ad hoc query scenario, the content of the user's query in this scenario is flexible, the calculation complexity is lower than that of offline batch tasks, and the results need to be obtained in a short time. Since the cold start speed of the computing engine in Hive is slow, in the ad hoc query scenario, the user's query task may only take 10 seconds to complete, but the cold start time of the computing engine takes minutes, making the existing computing engine not well suited for ad hoc query scenarios.
[0053] Therefore, this application creates a reusable computing engine and stores the engine information when creating the computing engine in the storage system, so that when HiveServer2 receives a task request initiated by the same client in the future, it can search for the corresponding historical engine information according to the client information, and then determine the computing engine that is suitable for the client. Then the computing engine is connected to HiveServer2, so that HiveServer2 can send the task request in the session to the computing engine for task processing.
[0054] It can be seen that the present application obtains the client information carried in the target task request in response to the target task request received from the client through the service component set in the server side; obtains the historical engine information corresponding to the client information in the engine registration information stored in the storage system of the server side; wherein the historical computing engine corresponding to the historical engine information has processed the task request of the client; if the acquisition fails, it indicates that the historical computing engine corresponding to the client does not exist in the server yet, and therefore, the service component is required to perform engine creation processing based on the target session to obtain a newly created computing engine to process the target task request sent by the target session; if the acquisition is successful, it indicates that the historical computing engine corresponding to the client exists in the server, and the service component connects the historical computing engine to the client through the target session, so that the historical computing engine can process the target task request sent by the client through the target session; thereby, the computing engine can be reused, the startup time of the computing engine is saved, and the task processing efficiency is improved.
[0055] Based on the above embodiment, the embodiment of the present application describes the steps of sending the target task request to the historical computing engine for task processing. Among them, the server receives the target task request through the target session, and the historical engine information includes the engine network address and the engine port. Specifically, the method of this embodiment includes the following steps:
[0056] Acquire the distributed lock corresponding to the client based on the target session; in response to the successful acquisition of the distributed lock, determine the corresponding historical computing engine based on the engine network address and the engine port; send the target task request received through the target session to the historical computing engine for task processing.
[0057] In combination with the above-mentioned embodiment, if the engine information of the historical computing engine corresponding to the client is successfully obtained, the service component HiveServer2 can connect the client to the historical computing engine through the target session based on the engine network address and engine port of the historical computing engine, and process the target task request sent by the client through the target session based on the historical computing engine.
[0058] Among them, distributed lock refers to a way to control the synchronous access to shared resources between distributed systems, mainly through mutual exclusion mechanism to prevent interference with each other to ensure consistency. In this application, the distributed lock service can set a distributed lock for each client. Taking a client (user 0) as an example, when the service component HiveServer2 processes the session of the client, it will first process the session with the distributed lock. The session without the distributed lock can only be processed after waiting for other sessions to release the distributed lock and acquiring the distributed lock itself. Each client corresponds to a distributed lock, which can not only avoid the problem of resource contention between multiple sessions of the client, but also control a client to create a computing engine, so as to realize the reuse of one computing engine for multiple sessions of the client.
[0059] For example, Figure 3 There are the first session (session 0), the second session (session 1) and the third session (session 2) of user 0, wherein the client is connected to service component-0 through session 0, to service component 1 through session 1, and to service component n through session 2. If service component 0 obtains the distributed lock of the client first, it indicates that service component 0 can process the task request in session 0 first, service component 0 creates a new computing engine and connects with the client through session 0, and then service component 0 can release the distributed lock of the client occupied by session 0. At this time, if service component 1 obtains the released distributed lock of the client, service component 1 can process session 1 again, find the historical computing engine corresponding to the client based on the task request from session 1, and complete the connection with the historical computing engine, and then service component 1 can release the distributed lock of the client occupied by session 1. Similarly, if service component n continues to obtain the distributed lock of the client that has been released, service component n can process session 2 of the client again, find the historical computing engine corresponding to the client based on the task request from session 2, and complete the connection with the historical computing engine. Then service component n can release the distributed lock occupied by session 2 so that other sessions can obtain the distributed lock to process tasks. The session with the distributed lock is processed first to avoid interference from other sessions, and at the same time, it also ensures that different sessions initiated by the same client can be connected to the same computing engine by the service component under the restriction of the distributed lock, realizing the reuse of the computing engine.
[0060] Exemplarily, in the specific implementation process of the present application, after the first session (session 0) and HiveServer2-0 establish a connection, HiveServer2-0 first needs to obtain the distributed lock of the client from the distributed lock service when processing the first session; if it can be obtained, it indicates that HiveServer2-0 can process session 0 initiated by the client; if it cannot be obtained, it needs to wait. Among them, the distributed lock service may include but is not limited to preset distributed lock services such as Zookeeper, Hbase, ETCD, etc., which are not limited here. In addition, if there are other sessions (such as session 1, session 2, etc.) at present, the HiveServer2 components corresponding to other sessions will also try to obtain the distributed lock of the client. The service component corresponding to which session obtains the distributed lock of the client first will execute the session first. However, if the HiveServer2-0 component responsible for processing session 0 has already obtained the distributed lock of the client, the other sessions of the client need to wait until the HiveServer2-0 component releases the distributed lock it has obtained before continuing to try to obtain the distributed lock corresponding to the client. This avoids interference from other sessions during the processing of session 0.
[0061] Similarly, in another exemplary embodiment, the steps before obtaining the engine information corresponding to the client based on the second session at least include: based on the second task request initiated by user 0, correspondingly creating session 1 (second session) between user 0 and the server, and obtaining a service address (HiveServer2-1) of a certain HiveServer2 through the load balancing gateway to establish a connection with session 1. Then, HiveServer2-1 will first try to obtain the distributed lock corresponding to the client, and wait if the acquisition fails; if the acquisition succeeds, HiveServer2-1 will then obtain the engine information of the computing engine previously created by the client from the storage system.
[0062] Therefore, HiveServer2-1 can connect to the corresponding historical computing engine based on the acquired historical engine information, so that the client can connect to HiveServer2-1 and the historical computing engine through the second session to avoid interference from other sessions. Subsequently, the historical computing engine can process the task request (pending task) issued by the client through the second session, and reuse the computing engine created based on the first session in the processing of the second session, reducing the process of destroying, recycling and recreating the computing engine in the traditional method, and improving the task processing efficiency.
[0063] It can be further explained that if the client sends more task requests, it can continue to create more sessions. For example, if user 0-session 2 is created (hereinafter referred to as the third session for ease of explanation), the timing of the third session can be after the second session. Therefore, if the third session is to be connected to the computing engine of the client, HiveSever2-2 needs to try to obtain the distributed lock when processing the third session; if the second session has not released the distributed lock, the third session cannot be connected to the computing engine for the time being; if the second session has released the distributed lock, and HiveSever2-2 obtains the distributed lock and successfully sets the third session, then HiveSever2-2 can similarly obtain the engine information of the computing engine used by the client from the storage system.
[0064] Based on the above embodiment, the present embodiment of the application describes the steps of creating a new computing engine for processing target task requests. Among them, the server receives the target task request through the target session. Specifically, the method of this embodiment includes the following steps:
[0065] A distributed lock corresponding to the client is obtained based on the target session; in response to successful acquisition of the distributed lock, a new computing engine corresponding to the client information is created.
[0066] In the above-mentioned embodiment, it is mainly explained that when there is a historical computing engine of the client in the server, if a target task request sent by the client through the target session is received, the corresponding HiveSever2 service component will find the historical computing engine and continue to connect, thereby realizing that the HiveSever2 service component will send the target task request sent by the client through the target session to the historical computing engine for processing.
[0067] This embodiment mainly explains that when there is no historical calculation engine of the client in the server, if a target task request sent by the client through the target session is received, a new calculation engine needs to be created to process the target task request.
[0068] Specifically, the process of creating a computing engine may at least include: the service component will try to obtain the distributed lock corresponding to the client based on the target session it is responsible for. If the distributed lock corresponding to the client can be obtained, it means that the target session can be processed, otherwise the target session needs to wait. After the service component responsible for processing the target session obtains the distributed lock corresponding to the client, it can apply for computing resources in the server to create a new computing engine corresponding to the client information.
[0069] Based on the above embodiment, the present embodiment of the application describes the steps after creating a new computing engine corresponding to the client information. Specifically, the method of this embodiment includes the following steps:
[0070] Each engine component in the newly created computing engine is started; the engine component includes an engine registrar; in response to each engine component being successfully started, the distributed lock of the client is released.
[0071] The process of creating a new computing engine includes creating various engine components in the new computing engine, and these engine components need to be started before the new computing engine can be officially put into use. After all engine components of the new computing engine are successfully started, it means that the service component HiveSever2 has successfully connected to the computing engine, and then the service component HiveSever2 can release the distributed lock it occupies.
[0072] Based on the above embodiment, the embodiment of the present application describes the steps of starting each engine component in the newly created computing engine and the steps of using the newly created computing engine to process the target task request. Specifically, the method of this embodiment includes the following steps:
[0073] Start a registration request receiver in the newly created computing engine; send an engine registration request corresponding to the target session to the registration request receiver so that the registration request receiver creates context resources for the target session; send a target task request to the newly created computing engine so that the newly created computing engine uses the context resources of the target session to process the target task request.
[0074] For example, reference may be made to Figure 4 As shown, Figure 4 It is a schematic diagram of the main modules of an exemplary computing engine driver in the task processing method of the present application, that is, a brief schematic diagram of the engine components in the computing engine of the present application. Among them, the engine components may include but are not limited to an engine registrar, a registration request receiver, a session manager, an engine information reporter, etc., which will not be repeated here. Among them, the engine registrar is mainly used to register the service component HiveSever2 in the computing engine, complete the connection between the service component and the computing engine, and then realize the connection between the computing engine and the session responsible for the service component; and the registration request receiver is mainly used to receive the engine registration request of HiveServer2 and perform registration processing (which may be based on SocketServer, HTTPServer, NettyServer, RPC and other any multi-system data interaction schemes to achieve); the session manager is mainly used to manage and the session context resources corresponding to the session connected to the computing engine, and the engine information reporter is mainly used to associate the engine information of the computing engine with the corresponding client information and report and store it in the storage system.
[0075] Specifically, when the computing engine of the present application is started, each engine component in the computing engine is started at the same time. For example, the registration request receiver is started to receive the engine registration request corresponding to the target session initiated by the service component HiveServer2; after the registration request receiver receives the engine registration request, it creates a set of session-isolated context resources for the target session (such as Figure 4 The computing engine can use the session context resources in the session manager to execute the computing tasks (task requests) sent by the corresponding session, so that multiple sessions of the same client can share the driver of the same computing engine, but the computing processes of different sessions are isolated from each other. After these engine components are ready, use Figure 4 The engine registrar in establishes a connection with HiveServer2, at which time HiveServer2 will release the distributed lock (for example, the engine registrar connects to HiveServer2-0 of the first session, and HiveServer2-0 releases the distributed lock of the first session) so that other sessions can obtain the distributed lock and continue to be processed. As a result, the computing engine is successfully created and started based on user 0-session 0, and the SQL query task request sent by the client through the first session will be sent by HiveServer2-0 to the corresponding computing engine for task execution processing, and the task processing result will also be returned to the client through HiveServer2-0 via the first session. It should be noted that the above example is a relatively main engine component startup process. In the specific process of starting the engine component, it can also include the startup of other components, such as the startup of the engine termination checker, the engine global context and other components, which will not be repeated here.
[0076] For example, if the registration request receiver of the computing engine receives an engine registration request initiated by the service component (HiveServer2-0) responsible for processing the first session (session 0), a first context resource for the first session is created; the first context resource is used to execute the task request of the first session. If the registration request receiver of the computing engine receives a registration invitation request initiated by the service component (HiveServer2-1) responsible for processing the second session (session 1), a second context resource for the second session is created; the second context resource is used to execute the task request of the second session, which is not described in detail here.
[0077] In conjunction with the above-mentioned embodiment, if HiveServer2-1 subsequently successfully obtains the distributed lock corresponding to the client, it indicates that session 1 can currently normally obtain and use the computing resources (computing engine) of the server without being interfered with by other sessions. Therefore, the Hive service component HiveSever2-1 corresponding to the second session can send a registration request to the computing engine that has been previously created by the client for registration processing. For example, after HiveSever2-1 obtains the distributed lock of the second session, it can first create a registration request client (specifically a client of the type corresponding to the current registration service), initiate a registration request to the computing engine through the registration request client, and wait for the connection of the computing engine. Since the computing engine has been started and ready in the previous processing of the first session, the connection time between the second session and the computing engine is usually very short, thereby achieving rapid connectivity between the computing engine and the second session. Therefore, the computing engine can also quickly respond to SQL tasks initiated by the client based on the second session.
[0078] Based on the above embodiment, the embodiment of the present application adds a step after storing the correspondence between the client information and the newly created computing engine in the engine registration information. Specifically, the method of this embodiment includes the following steps:
[0079] In response to a current task request received from a client, a current session corresponding to the current task request is created, and the timing of the current task request is later than the target task request; a distributed lock corresponding to the client is acquired based on the current session; in response to successful acquisition of the distributed lock, historical engine information corresponding to the client information is acquired from the stored engine registration information; in response to successful acquisition of the historical engine information, the current task request received through the current session is sent to the historical computing engine corresponding to the historical engine information for task processing.
[0080] In conjunction with the above-mentioned embodiment, after the new computing engine is created, the target task request (equivalent to the first task request) transmitted by the target session can be processed. If a current task request (equivalent to the second task request) initiated by the client is received later, the computing engine of the client can be searched and reused through the current task request (equivalent to the method of using the historical computing engine in the above-mentioned embodiment). For the latest current task request, the new computing engine created based on the target task request has become a historical computing engine.
[0081] For example, the current session can be understood as the second session (session 1) in the above example. If you want to process the second session, HiveSever2-1 corresponding to the second session needs to obtain the distributed lock of the client. After the distributed lock is successfully obtained, HiveSever2-1 will obtain the historical engine information. After obtaining the engine information, a registration request is initiated to the computing engine, and the connection to the computing engine is waited. Since the computing engine is already started and ready, it can quickly respond to the query tasks initiated by the user.
[0082] The above mainly explains the process of establishing a connection between a new session initiated by the client and a ready computing engine, which enables the reuse of computing engines. According to actual tests, the process of connecting a new session to an existing computing engine only takes seconds or milliseconds, which greatly shortens the time consumption compared to the traditional method of creating a corresponding computing engine for each session. In this way, the time delay from establishing a connection to executing a query will be significantly reduced, greatly improving the effect of the Hive engine in ad hoc query scenarios.
[0083] Based on the above embodiment, the present embodiment of the application describes the steps of creating a new computing engine for processing target task requests. Specifically, the method of this embodiment includes the following steps:
[0084] An engine is created based on the target task request to obtain a new computing engine, which includes a session context resource for processing the target task request; the target task request is sent to the new computing engine, so that the new computing engine processes the target task request based on the session context resource.
[0085] In conjunction with the foregoing embodiments, in the computing engine creation process of the present application, the computing engine corresponding to the client will create session-isolated context resources for different sessions of the client, and different session context resources are used to process the task requests transmitted by the client through different sessions. The specific method can be referred to the explanation of the foregoing embodiments, which will not be repeated here.
[0086] Based on the above embodiment, the embodiment of the present application describes the steps after responding to the target task request received from the client. Specifically, the method of this embodiment includes the following steps:
[0087] Based on the target task request initiated by the client, a target session is established with the client; and matching service components are allocated to the target session based on the running load on the server.
[0088] In conjunction with the foregoing embodiments, when the server receives a task request initiated by a client, it will establish a session with the client. For a certain client, it may initiate one or more sessions to the server, such as session 0, session 1, session 2, session, etc. in the example of this application. For each session, it can first connect to the load balancing gateway on the server side, and then the load balancing gateway selects a service address of a service component suitable for processing the session to connect to the session based on the current running load of the server side (such as the service pressure of each service component in the Hiveserver2 cluster) (for example, select service component-0 to be responsible for processing session 0, for example, select service component-1 to be responsible for processing session 1, etc.).
[0089] Based on the above embodiment, the embodiment of the present application describes the steps after sending the target task request to the historical computing engine for task processing. Among them, the server receives the target task request through the target session. Specifically, the method of this embodiment includes the following steps:
[0090] In response to the disconnection between the target session and the historical computing engine, the disconnection duration is recorded; if the disconnection duration is greater than a first time threshold, resource release processing is performed on the working node in the historical computing engine; if the disconnection duration is greater than a second time threshold, resource release processing is performed on the historical computing engine; the second time threshold is greater than the first time threshold.
[0091] With reference to the foregoing embodiments, in the task processing method of the present application, from the perspective of the overall process, after the client successfully connects to the computing engine in the server through the initiated session, the SQL request can be sent to the corresponding computing engine through the session, and the computing engine processes the received SQL task to obtain the task processing result, and returns the task processing result to the client through the session.
[0092] It should be noted that, usually, after the client receives the returned task processing result, the client will actively disconnect the session connection with the server, or disconnect the connection with the server after the session has been unresponsive for a period of time. In the existing method, if a session is disconnected from the computing engine, the computing engine will be shut down and destroyed and the computing resources will be recycled.
[0093] In the task processing method of the present application, when a session of the client is disconnected, the computing engine will not be shut down immediately, but can continue to process task requests (pending tasks) sent by other sessions. For example, in the aforementioned example, user 0 is connected to the same computing engine with at least two sessions (including session 0 and session 1). Even if user 0 closes session 0, causing session 0 to be disconnected from the computing engine, the computing engine will not be recycled immediately, but can continue to process pending tasks sent by session 1. If user 0 creates a new session with HiveServer2 again, there is no need to wait for the cold start time of the computing engine. According to the aforementioned example, the computing engine currently used by user 0 is registered based on the current new session, so that the new session can be connected to the computing engine.
[0094] For example, if all sessions of the client are disconnected from the computing engine, it indicates that the computing engine has entered an idle state. In existing methods, resources of idle computing engines are usually recycled, but in this application, computing resources of computing engines in an idle state can be recycled in stages by setting time thresholds of different values to balance the engine startup efficiency and resource idleness, so as to avoid the client from initiating task requests to the server again in a short period of time and needing to recreate the computing engine.
[0095] Specifically, a first time threshold and a second time threshold may be set, and the second time threshold is greater than the first time threshold, for example, the first time threshold may be 8 hours, and the second time threshold may be 24 hours. When there is no connection between the client's session and the computing engine, that is, when all sessions of the client are disconnected from the computing engine, the disconnection duration is recorded. In response to the disconnection duration being greater than the first time threshold, the resource release processing is first performed on the working node in the computing engine (the driver may be temporarily retained), so that the computing resources occupied by the working node in the computing engine can be recycled to the cluster; if the disconnection duration is within the interval between the first time threshold and the second time threshold, the client initiates a new session request and needs to reconnect to the computing engine, then the computing resources of the working node may be applied for and bound to the driver of the computing engine, thereby obtaining a complete computing engine. Among them, the driver may be regarded as the master node in the computing engine, and the working node may be regarded as a slave node of the master node. It is understandable that if there is a reconnected session during the disconnection duration recording process, the disconnection duration needs to be reset.
[0096] Furthermore, in response to the disconnection duration being greater than the second time threshold and a computing engine still having no session connected to it, the engine information in the storage system is cleared and the driver program (master node) of the computing engine is shut down to completely release the computing resources of the computing engine for use by other clients in need that have a communication connection with the server.
[0097] Based on the above embodiment, the embodiment of the present application describes the steps after recording the disconnection duration in response to the disconnection between the target session and the history calculation engine. Specifically, the method of this embodiment includes the following steps:
[0098] In response to the disconnection duration being less than or equal to the first time threshold and receiving a current task request initiated by the client, a current session is established with the client; the current session is connected to the historical computing engine, and the current task request received based on the current session is sent to the historical computing engine, so that the historical computing engine processes the current task request.
[0099] In conjunction with the above-mentioned embodiment, if the only target session connected between the client and the server is disconnected, when the disconnection duration is less than or equal to the first time threshold, if a new current task request initiated by the client is received again, the current session between the client and the server is first established, and the current session of the client is connected to the historical computing engine corresponding to the client by searching for the historical computing engine in the above-mentioned embodiment. The service component responsible for processing the current session can then send the current task request issued by the client based on the current session to the corresponding historical computing engine, so that the historical computing engine processes the current task request. This significantly speeds up the speed at which the client and the computing engine reconnect, that is, improves the task processing efficiency.
[0100] In summary, through the task processing method of this application, the computing engine is usually started only when the client creates a session connection with the server for the first time. If the client creates other sessions during this period, it will quickly connect to the ready computing engine (driver). In addition, if the client no longer needs to use the computing engine, the resources of the working node of the computing engine will be released first. If the client recreates the session within the first time threshold, it can still quickly connect to the ready driver and apply for computing resources to build a working node. Only when the session is offline for a long time will the driver of the computing engine be completely shut down. As a result, each client in this method can use a different driver, ensuring resource isolation between different clients, and enhancing the stability of the SQL query engine, without the occurrence of Driver single point problems; there is no need to switch the HiveSQL architecture to the SparkSQL architecture to achieve the reuse of various Hive-supported computing engines, achieve the effect of accelerating ad hoc queries, and ensure the compatibility of the query system with HiveSQL. The business layer does not need to modify the usage method and habits, and is completely unaware of the underlying modifications, and the migration cost is almost zero; furthermore, compared with local session caching in HiveServer2 to achieve computing engine reuse, this method can make full use of the load balancing capabilities of the HiveServer2 cluster. In the event of a crash in any HiveServer2 node, the client can still reuse the original engine when recreating the session, greatly enhancing stability.
[0101] It should be further explained that the execution subject of the task processing method may be a task processing device, for example, the task processing method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the task processing method may be implemented by a processor calling a computer-readable instruction stored in a memory.
[0102] Figure 5 FIG. 1 is a block diagram of a task processing device shown in an exemplary embodiment of the present application. Figure 5 As shown, the exemplary task processing device 500 includes: a request receiving module 510, an engine information obtaining module 520, a task processing module 530 and an engine creating module 540. Specifically:
[0103] The request receiving module 510 is used to obtain the client information carried in the target task request in response to the target task request received from the client.
[0104] The engine information acquisition module 520 is used to acquire historical engine information corresponding to the client information from the stored engine registration information; wherein the historical computing engine corresponding to the historical engine information has processed the task request of the client.
[0105] The task processing module 530 is used to send the target task request to the history computing engine for task processing in response to successfully obtaining the history engine information.
[0106] The engine creation module 540 is used to create a new computing engine for processing the target task request in response to the failure to obtain historical engine information, use the new computing engine to process the target task request, and store the correspondence between the client information and the new computing engine in the engine registration information.
[0107] In this exemplary task processing device, a service component set in the server side responds to a target task request received from a client and obtains the client information carried in the target task request; obtains the historical engine information corresponding to the client information from the engine registration information stored in the storage system of the server side; wherein the historical computing engine corresponding to the historical engine information has processed the task request of the client; if the acquisition fails, it indicates that the historical computing engine corresponding to the client does not exist in the server for the time being, and therefore, the service component is required to perform engine creation processing based on the target session to obtain a newly created computing engine to process the target task request sent by the target session; if the acquisition succeeds, it indicates that the historical computing engine corresponding to the client exists in the server, and the service component connects the historical computing engine to the client through the target session, so that the historical computing engine can process the target task request sent by the client through the target session; thereby, the computing engine can be reused, the startup time of the computing engine is saved, and the task processing efficiency is improved.
[0108] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.
[0109] Among them, the functions of each module can be found in the task processing method embodiment, and will not be repeated here.
[0110] See also Figure 6 , Figure 66 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 600 includes a memory 601 and a processor 602, and the processor 602 is used to execute program instructions stored in the memory 601 to implement the steps in any of the above-mentioned task processing method embodiments. In a specific implementation scenario, the electronic device 600 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 600 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.
[0111] Specifically, the processor 602 is used to control itself and the memory 601 to implement the steps in any of the above-mentioned task processing method embodiments. The processor 602 can also be called a CPU (Central Processing Unit). The processor 602 may be an integrated circuit chip with signal processing capabilities. The processor 602 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 602 can be implemented by an integrated circuit chip.
[0112] In this exemplary electronic device, a service component set in the server side responds to a target task request received from a client and obtains the client information carried in the target task request; obtains the historical engine information corresponding to the client information from the engine registration information stored in the storage system of the server side; wherein the historical computing engine corresponding to the historical engine information has processed the task request of the client; if the acquisition fails, it indicates that the historical computing engine corresponding to the client does not exist in the server yet, and therefore, the service component is required to perform engine creation processing based on the target session to obtain a newly created computing engine to process the target task request sent by the target session; if the acquisition succeeds, it indicates that the historical computing engine corresponding to the client exists in the server, and the service component connects the historical computing engine to the client through the target session, so that the historical computing engine can process the target task request sent by the client through the target session; thereby, the computing engine can be reused, the startup time of the computing engine is saved, and the task processing efficiency is improved.
[0113] See also Figure 7 , Figure 7The computer-readable storage medium 710 stores program instructions 711 that can be executed by a processor, and the program instructions 711 are used to implement the steps in any of the above-mentioned task processing method embodiments.
[0114] In this exemplary storage medium, the service component set in the server side is controlled by running the program instructions in the storage medium, and in response to the target task request received from the client, the client information carried in the target task request is obtained; in the engine registration information stored in the storage system of the server side, the historical engine information corresponding to the client information is obtained; wherein, the historical computing engine corresponding to the historical engine information has processed the task request of the client; if the acquisition fails, it indicates that the historical computing engine corresponding to the client does not exist in the server yet, and therefore, the service component is required to perform engine creation processing based on the target session to obtain a newly created computing engine to process the target task request sent by the target session; if the acquisition is successful, it indicates that the historical computing engine corresponding to the client exists in the server, and the service component connects the historical computing engine to the client through the target session, so that the historical computing engine can process the target task request sent by the client through the target session; thereby, the computing engine can be reused, the startup time of the computing engine is saved, and the task processing efficiency is improved.
[0115] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0116] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.
[0117] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0118] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
Claims
1. A task processing method, characterized in that: The method is applied to the server side, and the method comprises: In response to a received target task request initiated by a client, obtaining client information carried in the target task request; Acquire historical engine information corresponding to the client information from the stored engine registration information; wherein the historical computing engine corresponding to the historical engine information has processed the task request of the client; In response to successfully acquiring the historical engine information, sending the target task request to the historical computing engine for task processing; In response to the target session initiating the target task request being disconnected from the historical computing engine, the disconnection duration is recorded; and computing resources of the historical computing engine are recycled in stages according to the disconnection duration and a preset time threshold; If the disconnection duration is longer than a first time threshold, resource release processing is performed on the working node in the historical computing engine; if the disconnection duration is longer than a second time threshold, resource release processing is performed on the historical computing engine; the second time threshold is greater than the first time threshold; In response to a failure in obtaining the historical engine information, a new computing engine for processing the target task request is created, the target task request is processed using the new computing engine, and the correspondence between the client information and the new computing engine is stored in the engine registration information.
2. The method according to claim 1, characterized in that The server receives the target task request through the target session, the historical engine information includes the engine network address and the engine port, and the target task request is sent to the historical computing engine for task processing, including: Acquire a distributed lock corresponding to the client based on the target session; In response to the distributed lock being acquired successfully, determining a corresponding historical computing engine based on the engine network address and the engine port; The target task request received through the target session is sent to the historical calculation engine for task processing.
3. The method according to claim 1, characterized in that The server receives the target task request through a target session, and the creating a new computing engine for processing the target task request includes: Acquire a distributed lock corresponding to the client based on the target session; In response to the distributed lock being acquired successfully, a new computing engine corresponding to the client information is created.
4. The method according to claim 3, characterized in that After creating a new computing engine corresponding to the client information, the method further includes: Starting each engine component in the newly created computing engine; In response to each engine component being started successfully, the distributed lock of the client is released.
5. The method according to claim 4, characterized in that The starting of each engine component in the newly created computing engine includes: Starting a registration request receiver in the newly created computing engine; Sending an engine registration request corresponding to the target session to the registration request receiver, so that the registration request receiver creates a context resource for the target session; The using the newly created computing engine to process the target task request includes: The target task request is sent to the newly created computing engine, so that the newly created computing engine processes the target task request by using the context resources of the target session.
6. The method according to claim 1, characterized in that After the response to the received target task request initiated by the client, the method further includes: Establishing a target session with the client based on the target task request initiated by the client; A matching service component is allocated to the target session based on the running load of the server.
7. The method according to claim 1, characterized in that After recording the disconnection duration in response to the target session and the historical computing engine being disconnected, the method further includes: In response to the disconnection duration being less than or equal to the first time threshold and receiving a current task request initiated by the client, establishing a current session with the client; The current session is connected to the historical computing engine, and a current task request received based on the current session is sent to the historical computing engine, so that the historical computing engine processes the current task request.
8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 7.
9. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Method, device and equipment for executing SQL (Structured Query Language) query and storage medium
CN116610760A
