Data processing method and related device

Abstract

The invention discloses a data processing method, which is applied to guarantee the independence of processing resources among different users using a database. In the data processing method, a connection request of a user for a database system is obtained by setting a connection processing module, and a thread group bound with the user is created for the user based on the connection request, so that each created thread group only processes a data operation request under a specific user; the independence of processing resources among different users is ensured, and mutual interference among different users is avoided. Besides, a load detection module in the database system can detect the load condition of the created thread groups, so that some thread groups without loads for a long time are deleted, the phenomenon that too many idle thread groups occur due to binding of the thread groups and the user is avoided, and it is guaranteed that the database system has a high resource utilization rate.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 20, 2023, with application number 202311368188.0 and invention name “A request processing method, device and computing device cluster”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of database technology, and in particular to a data processing method and related devices. Background Art

[0003] A database system is a system consisting of a database and its management software. It can process various data operation requests sent by users to the database and return the data operation results to the users. In commercial scenarios, database systems are generally provided to users in the form of leasing. That is, users become tenants of the database system through leasing and have the right to operate the database system during the leasing period.

[0004] Traditional database systems usually only serve a single tenant, which easily leads to idle resources and waste of resources. Based on this, current database systems tend to allow multi-tenant sharing, that is, multiple tenants can access the same database system at the same time, thereby improving resource utilization and reducing the cost of database system providers.

[0005] In a multi-tenant shared database system, a thread pool consisting of multiple worker threads is usually constructed to centrally process data query requests sent by different tenants. However, when data query requests from different tenants are centrally processed based on one thread pool, resource contention is likely to occur between different tenants, that is, the data query performance of one tenant may be disturbed by other tenants, resulting in unstable performance. Summary of the invention

[0006] The present application provides a data processing method, which can ensure the independence of processing resources between different users and avoid mutual interference between different users.

[0007] The first aspect of the present application provides a data processing method, which is applied to a database system. The data processing method includes: a connection processing module of the database system receives a first connection request, and the first connection request carries the first user identity information of a first user. That is, the first connection request is initiated by the first user, and the first connection request is used to request to establish a connection with the database system.

[0008] Then, the connection processing module creates a first thread group for the first user based on the first user identity information of the first user. The first thread group is bound to the first user, and the first thread group includes at least one working thread, and the first thread group is used to process data operation requests corresponding to the first user. That is, by establishing a binding relationship between the first thread group and the first user, at least one working thread in the first thread group can be used to process data operation requests from the first user, and will not process data operation requests from other users.

[0009] Secondly, the connection processing module receives a second connection request, and the second connection request carries the second user identity information of the second user. The connection processing module creates a second thread group for the second user based on the second user identity information of the second user. The second thread group is bound to the second user, and the second thread group includes at least one working thread, and the second thread group is used to process the data operation request corresponding to the second user. That is, the working threads in the second thread group are all used to process the data operation request from the second user, and will not process the data operation request of other users.

[0010] In addition, the load detection module of the database system detects the load conditions of the first thread group and the second thread group. If the load detection module detects that the duration of no load on the target thread group exceeds the target duration, the target thread group is deleted, and the target thread group is the first thread group or the second thread group. In other words, if the duration of no load on a thread group exceeds the target duration, it means that the user corresponding to the thread group has not accessed the database for a long time, then the load detection module will delete the thread group, thereby releasing the resources occupied by the thread group to ensure that the database system has a high resource utilization rate.

[0011] In this solution, a connection processing module is set to obtain the user's connection request for the database system, and a thread group bound to the user is created for the user based on the connection request, so that each created thread group only processes data operation requests under a specific user, ensuring the independence of processing resources between different users and avoiding mutual interference between different users. In addition, the load detection module in the database system can detect the load of the created thread group, thereby deleting some thread groups that have been unloaded for a long time, avoiding the phenomenon of too many idle thread groups due to binding the thread group to the user, and ensuring that the database system has a high resource utilization rate.

[0012] In a possible implementation, when the connection processing module receives the first connection request, there is no thread group bound to the first user in the database system. That is, when there is no thread group bound to a user in the database system, the connection processing module will automatically trigger the creation of a corresponding thread group for the user when it obtains the connection request of the user, thereby improving the efficiency of responding to the user's data operation request in the future.

[0013] In one possible implementation, the connection processing module receives a third connection request, wherein the third connection request carries the first user identity information, and the connection processing module establishes a binding relationship between a new connection established based on the third connection request and the first thread group, so that the first thread group can process data operation requests from the new connection.

[0014] That is to say, for different connections established based on the same user identity, these different connections can be bound to the same thread group, so that the same thread group can handle data operation requests from different connections of the same user, avoiding the creation of too many thread groups, reducing the resources occupied by the thread groups, and effectively improving the resource utilization of the database system.

[0015] In a possible implementation, when the load detection module detects that the average load of the thread group bound to the first user is greater than a first threshold, a new thread group bound to the first user is created.

[0016] That is, if the average load of one or more thread groups currently bound to the first user is high, the load detection module is responsible for creating a new thread group bound to the first user to alleviate the processing pressure of the existing thread group, improve the processing efficiency of the data operation request corresponding to the first user, and avoid the user having to wait too long when querying data in the database.

[0017] In a possible implementation, when the load detection module detects that the average load of the thread group bound to the first user is less than a second threshold and the number of thread groups bound to the first user is greater than 1, some thread groups corresponding to the first user are deleted.

[0018] That is to say, if a user corresponds to multiple thread groups and the average load of the multiple thread groups is not high, it means that the data operation requests of the user no longer require too many thread groups to process. Therefore, a thread group corresponding to the user can be deleted, so that the load originally borne by the deleted thread group can be distributed to other thread groups, thereby improving the utilization rate of the thread group.

[0019] In one possible implementation, the average load of the thread group is determined based on any one or more of a plurality of load indicators, where the plurality of load indicators include: the number of pending data operation requests in the thread group, the number of active working threads in the thread group, and the size of processing resources occupied by the thread group.

[0020] In a possible implementation, the load detection module creates a new working thread or deletes some of the original working threads in the first thread group based on the load condition of the first thread group. For example, when the average load of the working threads in the first thread group is high, the load detection module can create a new working thread in the first thread group; for another example, when the average load of the working threads in the first thread group is low, the load detection module can delete one or more working threads in the first thread group. In addition, in addition to the first thread group, the load detection module can also detect the load condition of other thread groups, and create a new working thread for the thread group or delete the original working thread based on the load condition of the thread group.

[0021] In this solution, by detecting the load of working threads in the thread group based on the load detection module, working threads are added or deleted from the thread group, which can improve the resource utilization of the thread group, ensure rapid response to user requests, and save the memory resources and processing resources of the database system as much as possible.

[0022] In a possible implementation, the monitoring module of the database system obtains data operation requests from different users, and based on the user corresponding to the obtained data operation request, transmits the obtained data operation request to the corresponding thread group. That is, the monitoring module is specifically used to monitor the data operation request sent by the user, and according to the user corresponding to the data operation request and the binding relationship between the user and the thread group, the data operation request is assigned to the corresponding thread group for processing.

[0023] That is to say, the monitoring module is actually used to monitor the established connections between the database system and the users, so as to obtain data operation requests from different users; then, based on the user corresponding to each acquired data operation request, the data operation request is passed to the thread group corresponding to the same user, so that each thread group processes the data operation request of the corresponding user. Moreover, by independently setting up a monitoring module outside the thread group to realize the monitoring of data operation requests, the monitoring module can be made unbound to the user. Compared with the function of monitoring data operation requests by the thread group, independently setting up a monitoring module outside the thread group can avoid setting up a dedicated monitoring thread in each thread group, thereby greatly reducing the number of threads in the thread group and reducing resource usage.

[0024] In a possible implementation, the first thread group further includes a request queue, which is used to store data operation requests, and at least one working thread in the first thread group is used to retrieve the data operation request from the request queue and process the retrieved data operation request.

[0025] In a possible implementation, the processing resources occupied by the first thread group and the second thread group are pre-configured. That is, in the thread groups created in the database system, the processing resources occupied by the thread groups bound to the same user are pre-configured. For example, for each thread group bound to a user, a certain central processing unit (CPU) resource can be pre-configured, so that different thread groups can occupy independent CPU resources, ensuring that CPU isolation can be achieved between different users using the database system, and ensuring that each user can have stable database query performance.

[0026] In a possible implementation, the connection processing module and the load detection module are both implemented by one or more threads.

[0027] According to a second aspect of the present application, there is provided a data processing device, which is applied to a database system and includes: a receiving module, which is used to receive a first connection request, the first connection request carries first user identity information of a first user; a processing module, which is used to create a first thread group for the first user based on the first user identity information of the first user, the first thread group is bound to the first user, and the first thread group includes at least one working thread, and the first thread group is used to process data operation requests corresponding to the first user; the receiving module is also used to receive a second connection request, the second connection request carries second user identity information of a second user; the processing module is also used to create a second thread group for the second user based on the second user identity information of the second user, the second thread group is bound to the second user, and the second thread group includes at least one working thread, and the second thread group is used to process data operation requests corresponding to the second user; the processing module is also used to detect the load conditions of the first thread group and the second thread group; if it is detected that the continuous duration of no load of the target thread group exceeds the target duration, the processing module is also used to delete the target thread group, the target thread group is the first thread group or the second thread group.

[0028] In a possible implementation, when the receiving module receives the first connection request, there is no thread group bound to the first user in the database system.

[0029] In one possible implementation, the receiving module is further used to receive a third connection request, which carries the first user identity information; the processing module is further used to establish a binding relationship between a new connection established based on the third connection request and the first thread group, so that the first thread group can process data operation requests from the new connection.

[0030] In a possible implementation manner, the processing module is further configured to create a new thread group bound to the first user when it is detected that the average load of the thread group bound to the first user is greater than a first threshold.

[0031] In a possible implementation, the processing module is further configured to delete some thread groups bound to the first user when it is detected that the average load of the thread groups bound to the first user is less than a second threshold and the number of thread groups bound to the first user is greater than 1.

[0032] In one possible implementation, the average load is determined based on any one or more of a plurality of load indicators, wherein the plurality of load indicators include: the number of pending data operation requests in the thread group, the number of active working threads in the thread group, and the size of processing resources occupied by the thread group.

[0033] In a possible implementation, the processing module is further configured to create a new working thread or delete some of the original working threads in the first thread group based on the load condition of the first thread group.

[0034] In a possible implementation, the processing module is further configured to obtain data operation requests from different users, and based on the users corresponding to the obtained data operation requests, transmit the obtained data operation requests to the corresponding thread group.

[0035] In a possible implementation, the first thread group further includes a request queue, the request queue is used to store data operation requests, and at least one working thread in the first thread group is used to retrieve the data operation request from the request queue and process the retrieved data operation request.

[0036] In a possible implementation, the processing resources occupied by the first thread group and the second thread group are pre-configured.

[0037] The third aspect of the present application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory. The processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster performs the method of the first aspect or any implementation of the first aspect. For the steps in each possible implementation of the first aspect performed by the computer device cluster, please refer to the first aspect for details, which will not be repeated here.

[0038] The fourth aspect of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method of any implementation manner of the first aspect.

[0039] A fifth aspect of the present application provides a circuit system, the circuit system includes a processing circuit, and the processing circuit is configured to execute a method in any implementation manner of the above-mentioned first aspect.

[0040] The sixth aspect of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute a method implemented in any one of the first aspects.

[0041] In a seventh aspect, the present application provides a chip system, which includes a processor for supporting an electronic device to implement the functions involved in any implementation of the first aspect. In a possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the electronic device. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0042] The beneficial effects of the second to seventh aspects mentioned above can be referred to the introduction of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the structure of a database system in the related art;

[0044] Figure 2 A schematic diagram of the structure of a database system provided in an embodiment of the present application;

[0045] Figure 3 Another schematic diagram of the structure of the database system provided in the embodiment of the present application;

[0046] Figure 4 A schematic diagram of the relationship between a connection established between a user and a database system provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of the relationship between a user and a thread group in a database system provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of the structure of a thread group provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of the relationship between a user and a thread group in a database system provided in an embodiment of the present application;

[0050] Figure 8A schematic diagram of a database system workflow provided in an embodiment of the present application;

[0051] Fig. 9 A schematic diagram of the correspondence between a connection and a monitoring thread provided in an embodiment of the present application;

[0052] Fig.10 A schematic diagram of a flow chart of a timed task thread performing a check on a thread group provided in an embodiment of the present application;

[0053] Fig.11 A schematic diagram of a flow chart of another timed task thread performing a check on a thread group provided in an embodiment of the present application;

[0054] Fig.12 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0055] Fig.13 A schematic diagram of the structure of a computing device 1300 provided in an embodiment of the present application;

[0056] Fig.14 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0057] Fig.15 A schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application;

[0058] Fig.16 A schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application are described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those of ordinary skill in the art that with the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0060] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the descriptions used in this way can be interchanged where appropriate, so that the embodiments can be implemented in a sequence other than that illustrated or described in the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps that appear in the present application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in this application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the present application.

[0061] To facilitate understanding, some technical terms involved in the embodiments of the present application are introduced below.

[0062] (1) Database

[0063] A database is a warehouse that organizes, stores, and manages data according to data structures. It is essentially a collection of large amounts of data that is stored in a computer for a long period of time, organized, shareable, and uniformly managed.

[0064] (2) Database System

[0065] A database system is a system composed of a database and its management software. Specifically, a database system is an ideal data processing system developed to meet the needs of data processing. It is also a software system that provides data for actual operational storage, maintenance and application systems. It is a collection of storage media, processing objects and management systems.

[0066] (3)Thread

[0067] A thread is the smallest unit that an operating system can schedule operations on. A thread is contained in a process and is the actual operating unit in the process. A thread refers to a single sequential control flow in a process. Multiple threads can run concurrently in a process, and each thread executes different tasks in parallel.

[0068] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the structure of a database system in the related art. Figure 1 As shown, the database system in the related art is a multi-tenant shared database system, such as Figure 1 The database system in is shared by tenants 1 to N. In addition, the database system includes a thread pool, which has multiple pre-built worker threads (i.e. Figure 1 Worker thread 1-worker thread M).

[0069] During the operation of the database system, tenants sharing the database system can send data query requests to the database system, and the database system will randomly assign the tenants' data query requests to the worker threads in the thread pool for processing. For a data query request sent by a tenant, if the worker thread assigned to process the data query request has a large number of data query requests from other tenants to be processed, then the response time for the data query request sent by the current tenant will be relatively long; if the worker thread assigned to process the data query request currently has no other data query requests to be processed, then the response time for the data query request sent by the current tenant will be relatively short.

[0070] Simply put, when a thread pool is used to centrally process data query requests from different tenants, there is a problem of resource contention between different tenants. That is, the data query performance of one tenant may be disturbed by other tenants, resulting in unstable performance.

[0071] In view of this, the embodiment of the present application provides a data processing method, which obtains the user's connection request for the database system by setting a connection processing module, and creates a thread group bound to the user for the user based on the connection request, so that each thread group created only processes the data operation request under a specific user, ensuring the independence of processing resources between different users and avoiding mutual interference between different users. In addition, the load detection module in the database system can detect the load of the created thread group, thereby deleting some thread groups that have been unloaded for a long time, avoiding the phenomenon of too many idle thread groups due to binding the thread group to the user, and ensuring that the database system has a high resource utilization rate.

[0072] Specifically, the data processing method provided in the embodiment of the present application is applied to a database system. Furthermore, the database system includes: a connection processing module and a load detection module. The connection processing module is used to obtain a connection request from a user, and create a corresponding thread group for the user based on the user's connection request, so that the data operation request subsequently sent by the user can be processed by the thread group corresponding to the user.

[0073] Exemplarily, the connection processing module may obtain a first connection request from a first user. The first connection request is used to request to establish a connection with a database system, and the first connection request may carry the first user identity information of the first user to indicate the user who sent the first connection request. Then, the connection processing module establishes a connection between the first user and the database system based on the first connection request. Furthermore, the connection processing module also creates a first thread group based on the user identity information of the first connection request (i.e., the first user identity information of the first user), and the created first thread group is bound to the first user.

[0074] Secondly, the connection processing module receives a second connection request, and the second connection request carries the second user identity information of the second user. The connection processing module creates a second thread group for the second user based on the second user identity information of the second user. The second thread group is bound to the second user, and the second thread group includes at least one working thread, and the second thread group is used to process the data operation request corresponding to the second user. That is, the working threads in the second thread group are all used to process the data operation request from the second user, and will not process the data operation request of other users.

[0075] For the thread group created by the connection processing module for any user, the thread group includes at least one working thread, and at least one working thread in the thread group is used to process the data operation request corresponding to the user bound to the thread group. For example, by establishing a binding relationship between the first thread group and the first user, at least one working thread in the first thread group can be used to process the data operation request from the first user, and will not process the data operation request from other users. Among them, the data operation request can be, for example, a request to perform operations such as querying, writing or deleting data in the database, and this embodiment does not specifically limit this.

[0076] That is to say, based on the connection processing module, different thread groups can be created in the database system for different users, so that each user's data operation request for the database can be processed based on the corresponding thread group, effectively avoiding the phenomenon of different users competing for processing resources, and ensuring that the data query performance of each user is stable. In general, in the presence of a connection processing module, it is not necessary to keep the user's thread group at all times in the database system. When there are a large number of users in the database system, thread groups corresponding to the users can be created on demand; users without data query services do not need to create thread groups for them, thereby saving the memory and processing resources of the database system. In addition, compared with forwarding the connection request to the user's thread group for processing, setting up a connection processing module separately can respond to the user's connection request more quickly.

[0077] In addition, a load detection module is continuously running in the database system, and the load detection module is used to detect the load of the thread groups created in the database system, and delete the thread groups that have been unloaded for a period of time exceeding a preset period. Among the created thread groups, multiple thread groups bound to different user identity information are respectively used to process data operation requests corresponding to different users. That is, for any created thread group in the database system, this created thread group will only be used to process the data operation requests corresponding to the user to which it is bound, and will not process the data operation requests corresponding to other users.

[0078] Exemplarily, the load detection module of the database system detects the load of the first thread group and the second thread group. If the load detection module detects that the duration of no load of the target thread group exceeds the target duration, the target thread group is deleted, and the target thread group is the first thread group or the second thread group.

[0079] That is, if a thread group has no load for a period of time exceeding the target time, it means that the user corresponding to the thread group has not accessed the database for a long time, and the load detection module will delete the thread group, thereby releasing the resources occupied by the thread group to ensure that the database system has a high resource utilization rate. The target time can be determined or adjusted according to the resource situation of the database system or the actual use of the database, for example, the target time is 5 hours, 10 hours or 1 day, and this embodiment does not limit the specific value of the preset time.

[0080] Optionally, in the thread groups created in the database system, the processing resources occupied by the thread groups bound to the same user are pre-configured. For example, the processing resources occupied by the first thread group and the second thread group mentioned above are pre-configured. The processing resources configured by the thread groups can be determined based on the total processing resources in the database system in actual scenarios and the number of users facing the database system, and are not specifically limited here.

[0081] For example, each thread group bound to a user can be pre-configured with a certain central processing unit (CPU) resource, so that different thread groups can occupy independent CPU resources, ensuring CPU isolation between different users using the database system and ensuring that each user can have stable database query performance.

[0082] For example, see Figure 2 , Figure 2 A schematic diagram of the structure of a database system provided in an embodiment of the present application. Figure 2 As shown, the database system includes a connection processing module, a load detection module, and one or more thread groups corresponding to different users. Figure 2 When a connection request is received from user 1 to user N in the database, the connection processing module can create a corresponding thread group for each user and establish a binding relationship between the thread group and the user identity information. For example, thread group 1 has a binding relationship with user 1, thread group 2 has a binding relationship with user 2...thread group N has a binding relationship with user N. In this way, the working threads in each thread group will process data operation requests from the users to which they are bound, thereby ensuring the independence of different users and avoiding resource competition. In addition, the load detection module in the database system will periodically detect the load of each thread group, thereby timely deleting some thread groups that have been unloaded for a long time and releasing the resources occupied by these idle thread groups.

[0083] Optionally, after the connection processing module obtains a connection request from a user, there may be multiple ways to trigger the connection processing module to create a thread group for the user identity corresponding to the connection request.

[0084] For example, in a possible implementation, when the connection processing module obtains the above-mentioned first connection request, if there is no thread group bound to the first user in the database system, the connection processing module can trigger the creation of a thread group bound to the first user. That is, when a user requests to establish a connection with the database system, if there is no thread group corresponding to the user in the database system, a corresponding thread group can be newly created for the user.

[0085] Alternatively, in another possible implementation, when the connection processing module obtains a connection request from a user, if the database has created a thread group corresponding to the user and the thread group corresponding to the user has a high load, then the connection processing module can create an additional new thread group for the user to cope with a large number of data operation requests from the user.

[0086] Optionally, when different user devices can send connection requests to the database system based on the same user identity information (for example, different employees within a company use the same account and password to log in to the database system on different devices), the connection processing module can bind the same thread group to different connections under the same user, so that the same thread group can process data operation requests from different connections.

[0087] Exemplarily, after the connection processing module receives the first connection request and creates a corresponding thread group for the first user corresponding to the first connection request, the connection processing module also receives a third connection request and establishes a connection between the first user and the database system based on the third connection request. The user identity information of the third connection request is the same as the user identity information of the first connection request, that is, both are the first user identity information. Since the connection processing module has already created a corresponding first thread group for the first user corresponding to the third connection request, the connection processing module can establish a binding relationship between the new connection established based on the third connection request and the first thread group created based on the first connection request. For example, assuming that the connection established by the connection processing module based on the first connection request is the first connection, and the connection established based on the third connection request is the second connection, then the first connection and the second connection actually correspond to the same user. Therefore, the thread group created by the connection processing module based on the user identity information corresponding to the first connection request can be simultaneously established with the first connection and the second connection, so that the thread group can process data operation requests from the first connection and the second connection.

[0088] Optionally, in order to enable each thread group to accurately process the data operation request of the corresponding user, the database system may further include: a monitoring module. The monitoring module obtains data operation requests from different users, and based on the user corresponding to the obtained data operation request, transmits the obtained data operation request to the corresponding thread group.

[0089] That is to say, the monitoring module is actually used to monitor the established connections between the database system and the users, so as to obtain data operation requests from different users; then, based on the user corresponding to each acquired data operation request, the data operation request is passed to the corresponding thread group, so that each thread group processes the data operation request of the corresponding user. In addition, by independently setting up a monitoring module outside the thread group to monitor the data operation request, the monitoring module can be unbound to the user. Compared with the function of monitoring data operation requests by the thread group, independently setting up a monitoring module outside the thread group can avoid setting up a dedicated monitoring thread in each thread group, thereby greatly reducing the number of threads in the thread group and reducing resource usage.

[0090] For ease of understanding, the specific implementation of the above-mentioned connection processing module, load detection module and monitoring module will be described in detail below with reference to specific examples.

[0091] See also Figure 3 , Figure 3 Another structural diagram of the database system provided in the embodiment of the present application. Figure 3 As shown, the database system includes a thread pool, and the functions completed by the above-mentioned connection processing module, load detection module and monitoring module can be implemented through the threads in the thread pool.

[0092] Specifically, the thread pool may include a connection thread, a group of default threads, a group of listener threads, a group of timer threads, and multiple thread groups. A group of default threads includes one or more default threads; a group of monitoring threads includes one or more monitoring threads; a group of timer task threads includes one or more timer task threads; and a thread group includes one or more working threads. Among them, the above-mentioned connection processing module can be implemented by using a connection thread and a group of default threads; the above-mentioned load detection module can be implemented by using a group of timer task threads; and the monitoring module can be implemented by using a group of monitoring threads. The number of default threads, the number of monitoring threads, the number of timer task threads, and the number of working threads in the thread group can all be automatically configured by the database system, or configured by the user based on the usage of the database system, and this embodiment does not specifically limit this.

[0093] The connection thread is used to monitor the service port number of the database system, that is, to monitor the connection request from the client used by the user, and forward the obtained connection request to the default thread for processing. When the user needs to use the database system, he will log in on the client and send a connection request to the database system.

[0094] The default thread is used to process the client's connection request, thereby establishing a connection between the client used by the user and the database system. In addition, the default thread will parse the user identity information of the established connection and bind the established connection to the thread group corresponding to the user to which it belongs. In some possible embodiments, the function of the default thread can also be replaced by a worker thread in the thread group. That is, after the connection thread obtains the connection request, it forwards the connection request to a worker thread in any thread group for processing.

[0095] The monitoring thread is used to monitor the data operation requests on the established connection between the database system and the client, and forward the data operation requests to the thread group corresponding to the user to which it belongs for processing based on the user identity information obtained for the data operation requests.

[0096] The thread group is used to process data operation requests from users, thereby performing operations such as data query for users.

[0097] The scheduled task thread is used to periodically check the load of each thread group in the thread pool, and perform elastic scaling of the thread group and the working threads in the thread group based on the load of the thread group.

[0098] See also Figure 4 , Figure 4 A schematic diagram of the relationship between a connection established between a user and a database system provided in an embodiment of the present application. Figure 4 As shown in (a) in the figure, the database system can assign a pair of account and password to each user who needs to use the database system. The user can use the account and password to log in to the database system on different clients and establish a connection with the database system. For example, the same user uses the same pair of account and password to log in to the database system through clients on a smartphone, a personal computer, a laptop, and a tablet computer and establish a connection with the database system, thereby establishing multiple connections between the database system and the same user. That is, the relationship between a user and a connection can be one-to-one or one-to-many.

[0099] like Figure 4As shown in (b) in the figure, the database system can also be leased to some large tenants for use, and these tenants have the authority to create and delete users. That is, one or more pairs of accounts and passwords can be created under one tenant for use by different users, and these users all belong to the same tenant. For example, assuming that a tenant is a company, the company can create different accounts and passwords for different departments for use by users in different departments. In addition, each user under the same tenant can use the account and password to log in to the database system on different clients and establish a connection with the database system. For example, the same user uses the same pair of account and password to log in to the database system through the client on a smartphone, a personal computer, a laptop, and a tablet computer and establish a connection with the database system, thereby establishing multiple connections between the database system and the same user. That is, the relationship between tenants and users is one-to-one or one-to-many, and the relationship between users and connections can be one-to-one or one-to-many.

[0100] See also Figure 5 , Figure 5 A schematic diagram of the relationship between a user and a thread group in a database system provided in an embodiment of the present application. Figure 5 As shown in (a) in FIG. 1 , for a user, the database system can create one or more thread groups for the user, and the one or more thread groups are bound to the user, such as Figure 5 In (a), one user corresponds to thread group 1-thread group n. Generally speaking, when a user establishes a connection with the database system for the first time, a thread group bound to the user will be created; when the user has many data operation requests and it is difficult to effectively process the user's data operation requests based on the existing thread group, a new thread group will be created for the user, so that one user may correspond to multiple thread groups.

[0101] like Figure 5 As shown in (b), when the database system is leased to some large tenants, one tenant may include one or more users, and each user may correspond to one or more thread groups. That is, the relationship between tenants and users is one-to-one or one-to-many, and the relationship between users and thread groups can be one-to-one or one-to-many.

[0102] It should be noted that Figure 4 and Figure 5It is introduced that a user can correspond to multiple connections and multiple thread groups respectively, but the multiple connections established between the user and the database system and the multiple thread groups corresponding to the user are not necessarily in a one-to-one correspondence, that is, each thread group corresponding to the user does not uniquely correspond to one connection. The thread group corresponding to the user may correspond to one or more connections established between the user and the database system. For example, there may be three connections established between the user and the database system, and the number of thread groups corresponding to the user may be one, that is, the one thread group corresponding to the user can be used to process data operation requests obtained from the three connections corresponding to the user; for another example, there may be five connections established between the user and the database system, and the number of thread groups corresponding to the user may be two, and the two thread groups corresponding to the user can both process data operation requests obtained from any of the five connections corresponding to the user.

[0103] The above introduces the connection between users and the database system and the relationship between thread groups. The following will introduce the composition of thread groups in the database system.

[0104] Specifically, in a database system, each thread group can be composed of a group of request queues and a group of worker threads. Among them, a group of request queues includes at least one request queue, and a group of worker threads includes at least one worker thread. The request queue is used to store data operation requests from the client; the worker thread is the consumer of the request queue, responsible for taking out the data operation request from the request queue and executing it. There can be many types of request queues, such as queues with priority or ordinary queues without priority. When processing data operation requests, data operation requests with higher priority can be stored in queues with higher priority so that they can be processed by the worker thread first; data operation requests with lower priority can be stored in queues with lower priority or ordinary queues without priority.

[0105] In addition, in a thread group, the number of request queues and the number of worker threads may be the same, and each worker thread uniquely corresponds to a request queue. That is, each worker thread takes out a data operation request from its corresponding request queue to execute. Optionally, in a thread group, the number of request queues and the number of worker threads may also be different, and each worker thread may take out a data operation request from any request queue to execute. In general, this embodiment does not limit the implementation method of the cooperation between the worker threads and the request queues in the thread group.

[0106] For example, see Figure 6 , Figure 6 A schematic diagram of the structure of a thread group provided in an embodiment of the present application. Figure 6As shown, the thread group includes request queue 1-request queue n and multiple worker threads. As consumers of data operation requests, multiple worker threads take data operation requests from request queue 1-request queue n for processing. When there is no data operation request to be processed in the request queue, the worker thread can enter an idle state.

[0107] Based on the above embodiments, it can be determined that one tenant corresponds to one or more users in the database system, and one user corresponds to one or more thread groups in the database system. Therefore, the corresponding relationship between users, thread groups and working threads can be as follows: Figure 7 shown. Figure 7 A schematic diagram of the relationship between a user and a thread group in a database system provided in an embodiment of the present application. Figure 7 In the , the correspondence between users and thread groups is 1:n, that is, one user corresponds to n thread groups, n is an integer greater than or equal to 1; the correspondence between thread groups and working threads is 1:m, that is, one thread group corresponds to m working threads, m is an integer greater than or equal to 1.

[0108] For ease of understanding, the following will describe in detail with reference to the accompanying drawings how each thread in the database system establishes a connection for a user and how to process a data operation request sent by the user based on the established connection.

[0109] See also Figure 8 , Figure 8 A schematic diagram of the workflow of a database system provided in an embodiment of the present application. Figure 8 As shown, the workflow of each thread in the database system includes the following steps 801-809.

[0110] Step 801: Obtain a connection request from a user.

[0111] In this embodiment, the connection thread in the database system will continuously monitor the service port number of the database system, that is, monitor the connection requests of each client. After obtaining the connection request sent by the user through the client, the connection thread will pass the obtained connection request to the default thread for processing. In the case where there are multiple default threads in the database system, the connection thread may select one of the default threads to pass the connection request. There are multiple ways for the connection thread to select the default thread, for example, the connection thread randomly selects a default thread from multiple default threads; or the connection thread divides the identifier of the connection request by the number of default threads and takes the remainder, thereby selecting the corresponding default thread based on the size of the remainder.

[0112] Optionally, the function of the connection thread may also be completed by the monitoring thread, that is, the monitoring thread monitors the connection request and the data operation request at the same time, which is not specifically limited in this embodiment.

[0113] Step 802: Establish a connection based on the connection request.

[0114] After the default thread obtains the connection request transmitted by the connection thread, a connection can be established between the client used by the user and the database system based on the connection request.

[0115] Step 803, determine whether the user to which the connection belongs has a corresponding thread group.

[0116] After the default thread successfully establishes a connection, it will parse the user identity information corresponding to the connection, that is, determine which user the created connection belongs to. Then, based on the user identity information corresponding to the connection, the default thread will determine whether the user to which the connection belongs has a corresponding thread group in the database system. For example, if the user to which the connection belongs is user 1, the default thread will detect whether there is a thread group bound to user 1 in the database system.

[0117] Step 804: If the user to which the connection belongs does not have a corresponding thread group, a thread group is created for the user to which the connection belongs.

[0118] When the user to which the connection belongs does not have a corresponding thread group, it means that the user has recently logged into the database system for the first time, so the default thread can create a thread group for the user to which the connection belongs, and bind the created thread group to the user to which the connection belongs.

[0119] Step 805: If the user to which the connection belongs has a corresponding thread group, it is determined whether the load of the thread group corresponding to the user to which the connection belongs is saturated.

[0120] When the user to which the connection belongs does not have a corresponding thread group, since the default thread establishes a new connection for the user, the user's data operation requests may increase, so the default thread determines whether the load of the thread group corresponding to the user to which the connection belongs is saturated.

[0121] Step 806: If the load of the thread group corresponding to the user to which the connection belongs is saturated, a new thread group is created for the user to which the connection belongs.

[0122] When the load of the thread group corresponding to the user to which the connection belongs is saturated, the existing thread group representing the user may have difficulty in coping with the subsequent increasing number of data operation requests, so the default thread is a newly created thread group for the user to which the connection belongs.

[0123] Step 807: If the load of the thread group corresponding to the user to which the connection belongs is not saturated, the connection is bound to the thread group corresponding to the user.

[0124] When the load of the thread group corresponding to the user to which the connection belongs is not saturated, the default thread will no longer create a new thread group for the user, but will bind the newly established connection to the thread group corresponding to the user so that the data operation request obtained from the connection can be processed by the thread group corresponding to the user.

[0125] Step 808: Obtain a data operation request from the established connection.

[0126] After the default thread completes the connection establishment based on the connection request, the default thread will assign the established connection to the monitoring thread, and the monitoring thread will monitor the data operation request on the connection. In the case where there are multiple monitoring threads in the database system, the default thread can select one of the monitoring threads to assign the connection. There are many ways for the default thread to select a monitoring thread, for example, the default thread randomly selects a monitoring thread from multiple monitoring threads; or the default thread divides the connection identifier by the number of monitoring threads and takes the remainder, thereby selecting the corresponding monitoring thread based on the size of the remainder.

[0127] The monitoring thread is responsible for monitoring data operation requests on the connection. When a data operation request comes in on the connection, the monitoring thread forwards the data operation request to the thread group corresponding to the user to which the connection belongs.

[0128] Specifically, there are many ways for the monitoring thread to monitor data operation requests, for example, the common epoll_wait() function can be used to implement monitoring. The number of monitoring threads can be configured by the user or automatically configured by the database system. It should be noted that one connection can only correspond to one monitoring thread; conversely, one monitoring thread can monitor multiple connections at the same time, as shown in 9. Fig. 9 A schematic diagram of the correspondence between a connection and a monitoring thread provided in an embodiment of the present application. Fig. 9 As shown, each connection established between a user and the database system uniquely corresponds to a monitoring thread, and a monitoring thread can monitor data query requests on one or more connections.

[0129] Step 809: Allocate the data operation request to the corresponding thread group for processing.

[0130] After the monitoring thread obtains the data operation request from the monitored connection, it can assign the data operation request to the corresponding thread group for processing according to the user to which the monitored connection belongs, that is, store the data operation request in the request queue of the corresponding thread group.

[0131] In the case that there are multiple thread groups corresponding to the user to which the connection belongs, the monitoring thread may select to assign the data operation request to one of the thread groups. There are multiple ways for the monitoring thread to select the thread group, for example, the monitoring thread randomly selects a monitoring thread from multiple thread groups; or the monitoring thread obtains a random number, divides the random number by the number of thread groups and takes the remainder, thereby selecting the corresponding thread group based on the size of the remainder.

[0132] When the thread group processes the user's data operation request, the worker thread in the thread group takes a data operation request from the request queue and processes it. When there is no data operation request in the thread group's request queue, the worker thread can enter the idle state.

[0133] The above introduces how each thread in the database system in this embodiment establishes a connection for a user and how to process a data operation request sent by the user based on the established connection. The following will introduce in detail how a group of timed task threads in the database system used to implement a load detection module can achieve elastic scaling of the thread group and the elastic scaling of the working threads in the thread group.

[0134] During the operation of the load detection module, the load detection module will periodically detect the load conditions of the thread groups that have been created in the database system, and delete the thread groups that have been unloaded for more than a certain period of time, thereby releasing the resources occupied by the idle thread groups and ensuring that the database system has a high resource utilization rate.

[0135] Optionally, the load detection module in the database system may create a new thread group bound to the first user when it is detected that the average load of the thread group bound to the first user is greater than a first threshold. That is, if the average load of one or more thread groups currently bound to the first user is high, the load detection module is responsible for creating a new thread group bound to the first user to relieve the processing pressure of the existing thread groups, improve the processing efficiency of the data operation request corresponding to the first user, and avoid the user having to wait too long when querying data in the database.

[0136] The average load of a thread group may refer to the average value of the load of one or more thread groups. The load of a thread group may be measured by one or more load indicators, such as the number of data operation requests waiting to be processed in the request queue of the thread group, the idleness of the request queue in the thread group, the number of active working threads in the thread group, and the size of the processing resources occupied by the thread group. This embodiment does not specifically limit how to determine the load of a thread group. For example, the load of a thread group is measured by the idle rate of the request queue in the thread group. The higher the idle rate of the request queue in the thread group, the lower the load of the thread group; the lower the idle rate of the request queue in the thread group, the higher the load of the thread group. When the average idle rate of the request queue in the thread group bound to the first user identity information is lower than 40%, it means that the average load of the thread group is greater than the first threshold, so a new thread group can be created.

[0137] Optionally, the load detection module in the database system may also delete some thread groups corresponding to the first user when it is detected that the average load of the thread groups bound to the first user is less than a second threshold and the number of thread groups bound to the first user is greater than 1.

[0138] That is to say, if a user corresponds to multiple thread groups and the average load of the multiple thread groups is not high, it means that the data operation requests of the user no longer require too many thread groups to process. Therefore, a thread group corresponding to the user can be deleted, so that the load originally borne by the deleted thread group can be distributed to other thread groups, thereby improving the utilization rate of the thread group.

[0139] Optionally, in addition to elastically scaling the thread group, the load detection module may also create a new working thread in a single thread group or delete some of the original working threads based on the load of a single thread group in the created thread group. Taking the first thread group mentioned above as an example, the load detection module may create a new working thread in the first thread group or delete some of the original working threads based on the load of the first thread group.

[0140] For example, when the average load of the working threads in the thread group is high, the load detection module can create a new working thread in the thread group; for another example, when the average load of the working threads in the thread group is low, the load detection module can delete one or more working threads in the thread group.

[0141] In this solution, by detecting the load of working threads in the thread group based on the load detection module, working threads are added or deleted from the thread group, which can improve the resource utilization of the thread group, ensure rapid response to user requests, and save the memory resources and processing resources of the database system as much as possible.

[0142] It should be noted that, in this embodiment, the load detection module can be implemented by using one or more timed task threads. When there are fewer thread groups created in the database system, one timed task thread can be used to implement load detection of the thread group; when there are more thread groups created in the database system, multiple timed task threads can be used to implement load detection of the thread group, and each timed task thread is responsible for managing a part of the thread groups. For example, based on the number of users who have established connections with the database system, the thread groups corresponding to the users are evenly distributed to multiple timed task threads, or the thread groups corresponding to active users are evenly distributed to multiple timed task threads.

[0143] For example, see Fig.10 , Fig.10 A schematic diagram of a flow chart of a timed task thread performing a check on a thread group provided in an embodiment of the present application. Fig.10 As shown in the figure, the scheduled task thread will periodically check the thread group it is responsible for, and the inspection task mainly includes two aspects. On the one hand, it checks the load of the thread group under each user, so as to achieve elastic scaling of the thread group under the same user; on the other hand, it checks the load of the working threads within the thread group, so as to achieve elastic scaling of the working threads within a single thread group. Fig.10 In the process of checking the thread group by the scheduled task thread, the steps mainly include the following steps.

[0144] Step 1: After starting to execute the thread group check, determine whether all users that the current scheduled task thread is responsible for have been checked.

[0145] If all users that the current scheduled task thread is responsible for have been checked, the check is ended; if all users that the current scheduled task thread is responsible for have not been checked, an unchecked user is taken out to perform the thread group check.

[0146] Step 2: Determine whether the currently retrieved user has not logged in for a long time.

[0147] If a user has not logged in for a long time, the thread group corresponding to the user must be unloaded for a long time. Therefore, if the request queue in the thread group corresponding to the user continues to be unloaded for longer than the preset time, it can be considered that the user has not logged in for a long time. If the user has not logged in for a long time, the scheduled task thread can delete the thread group corresponding to the user and continue to check the next user.

[0148] Step 3: If the user has not been logged in for a long time, check whether the thread group corresponding to the user is in an idle state.

[0149] If the thread group corresponding to the current user has no load (that is, there is no data operation request to be processed in the request queue), it means that the thread group corresponding to the user is in an idle state. Then, if the thread group corresponding to the user is in an idle state and the number of thread groups corresponding to the user is greater than 1, a thread group corresponding to the user can be deleted to release the memory resources and processing resources occupied by the idle thread group.

[0150] Step 4: When the thread group corresponding to the user is not in an idle state, determine whether the thread group corresponding to the user is busy.

[0151] When the thread group corresponding to the user is busy, if the number of thread groups corresponding to the user has not reached the upper limit, a new thread group can be created for the user, thereby reducing the load of the existing thread group corresponding to the user. In addition, when the thread group corresponding to the user is not busy, the average load of the thread group corresponding to the user is low, and the number of thread groups corresponding to the user is greater than 1, a thread group corresponding to the user can be deleted.

[0152] The method of determining whether the thread group corresponding to the user is busy can be determined based on one or more load indicators, such as the number of data operation requests waiting to be processed in the request queue of the thread group, the idleness of the request queue in the thread group, the number of active working threads in the thread group, and the size of the processing resources occupied by the thread group. For example, taking the idle rate of the request queue in the thread group as an example, when the average idle rate of the request queue in the thread group corresponding to the user is lower than 40%, it means that the thread group corresponding to the user is busy, so a new thread group can be created for the user; when the average idle rate of the request queue in the thread group corresponding to the user is greater than 80%, it means that the thread group corresponding to the user is not busy and has a low load, so a corresponding thread group can be deleted for the user. The average idle rate of the request queue in the thread group is obtained by averaging the idle rates of each request queue in the thread group. The idle rate of the request queue refers to the ratio between the number of data operation requests currently stored in the request queue and the maximum number of requests that the request queue can store.

[0153] Step 5, determine whether the thread group of the current user has been checked.

[0154] If the thread group of the current user has been checked, continue to check the thread groups of other users; if the thread group of the current user has not been checked, take out a thread group of the current user for checking.

[0155] Step 6: Determine whether the thread group taken out from the current user is in an idle state.

[0156] If the thread group taken out from the current user is in an idle state and there are idle working threads in the thread group, one working thread in the thread group can be deleted until there are no idle working threads in the thread group. The minimum number of working threads in the thread group can be 0.

[0157] Specifically, a method for determining whether the thread group is in an idle state may be to check whether a request queue in the thread group has a pending data operation request. If the request queue in the thread group has a pending data operation request, it means that the thread group is not in an idle state; if the request queue in the thread group does not have a pending data operation request, it means that the thread group is in an idle state.

[0158] Step 7: If the thread group taken out from the current user is not idle, it is further determined whether the thread group taken out is in a busy state.

[0159] Specifically, there are many ways to determine whether a thread group is in a busy state, such as determining the average idle rate of the request queues in the thread group to determine whether the thread group is in a busy state. When the average idle rate of the request queues in the thread group is low, it can be considered that the thread group is heavily loaded and the thread group is in a busy state; when the average idle rate of the request queues in the thread group is not low, it can be considered that the thread group is loaded low and the thread group is not in a busy state. For another example, if the request queue of the thread group is not empty, and during the interval between the two checks of the scheduled task thread, no new request in the thread group has been processed, it means that the thread group is in a busy state.

[0160] Step 8: If the currently retrieved thread group is in a busy state, determine whether there is an idle working thread in the thread group.

[0161] If there are idle worker threads in the thread group, the idle worker threads can be activated to make more worker threads available for use; if there are no idle worker threads in the thread group and the number of worker threads has not reached the upper limit, a new worker thread can be created in the thread group.

[0162] It should be noted that the upper limit of the number of working threads in a thread group may be configured by the user or by the database system itself, and this embodiment does not specifically limit this.

[0163] It is understandable that in Fig.10 In the checking process shown, the thread group corresponding to each user is first checked cyclically to determine whether the thread group corresponding to the user needs to be increased or decreased, and then the working threads in each thread group corresponding to the user are checked to determine whether the working threads in each thread group corresponding to the user need to be increased or decreased. Fig.10The loops in the checking process shown are nested, that is, all users are checked in a loop first, and after checking whether the thread group of the user needs to perform elastic scaling, it is checked whether the working threads in the thread group of the user perform elastic scaling.

[0164] See also Fig.11 , Fig.11 Another flow chart of a timed task thread checking a thread group provided in an embodiment of the present application. Fig.11 As shown, in a possible implementation, the scheduled task thread may first cyclically check the thread groups corresponding to all users, determine whether the thread group corresponding to each user needs to be increased or decreased, and then cyclically check whether the working threads in each thread group need to be increased or decreased. That is, in the checking process of the scheduled task thread, first check whether the thread groups of all users need to perform elastic scaling, and then cyclically check whether the working threads in each thread group need to perform elastic scaling.

[0165] The above introduces a data processing method and a database system provided in an embodiment of the present application. The following will introduce a device for implementing the workflow of the above database system.

[0166] See also Fig.12 , Fig.12 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Fig.12 As shown, the data processing device is applied to a database system, and specifically includes: a receiving module 1201, which is used to receive a first connection request, the first connection request carries the first user identity information of the first user; a processing module 1202, which is used to create a first thread group for the first user based on the first user identity information of the first user, the first thread group is bound to the first user, and the first thread group includes at least one working thread, and the first thread group is used to process a data operation request corresponding to the first user; the receiving module 1201 is also used to receive a second connection request, the second connection request carries the second user identity information of the second user; the processing module 1202 is also used to create a second thread group for the second user based on the second user identity information of the second user, the second thread group is bound to the second user, and the second thread group includes at least one working thread, and the second thread group is used to process a data operation request corresponding to the second user; the processing module 1202 is also used to detect the load conditions of the first thread group and the second thread group; if it is detected that the continuous duration of the target thread group without load exceeds the target duration, the processing module 1202 is also used to delete the target thread group, and the target thread group is the first thread group or the second thread group.

[0167] In a possible implementation, when the receiving module 1201 receives the first connection request, there is no thread group bound to the first user in the database system.

[0168] In one possible implementation, the receiving module 1201 is also used to receive a third connection request, which carries the first user identity information; the processing module 1202 is also used to establish a binding relationship between the new connection established based on the third connection request and the first thread group, so that the first thread group can process data operation requests from the new connection.

[0169] In a possible implementation, the processing module 1202 is further configured to create a new thread group bound to the first user when it is detected that the average load of the thread group bound to the first user is greater than a first threshold.

[0170] In a possible implementation, the processing module 1202 is further configured to delete some thread groups bound to the first user when it is detected that the average load of the thread groups bound to the first user is less than a second threshold and the number of thread groups bound to the first user is greater than 1.

[0171] In one possible implementation, the average load is determined based on any one or more of a plurality of load indicators, wherein the plurality of load indicators include: the number of pending data operation requests in the thread group, the number of active working threads in the thread group, and the size of processing resources occupied by the thread group.

[0172] In a possible implementation, the processing module 1202 is further configured to create a new working thread or delete some of the original working threads in the first thread group based on the load condition of the first thread group.

[0173] In a possible implementation, the processing module 1202 is further configured to obtain data operation requests from different users, and based on the users corresponding to the obtained data operation requests, transmit the obtained data operation requests to the corresponding thread group.

[0174] In a possible implementation, the first thread group further includes a request queue, the request queue is used to store data operation requests, and at least one working thread in the first thread group is used to retrieve the data operation request from the request queue and process the retrieved data operation request.

[0175] In a possible implementation, the processing resources occupied by the first thread group and the second thread group are pre-configured.

[0176] The receiving module 1201 and the processing module 1202 can be implemented by software or hardware. For example, the implementation of the processing module 1202 is described below by taking the processing module 1202 as an example. Similarly, the implementation of the acquisition module 1201 can refer to the implementation of the processing module 1202.

[0177] Processing module 1202 is an example of a software functional unit, and processing module 1202 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, processing module 1202 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0178] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0179] The processing module 1202 is taken as an example of a hardware functional unit, and the processing module 1202 may include at least one computing device, such as a server, etc. Alternatively, the processing module 1202 may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0180] The multiple computing devices included in the processing module 1202 can be distributed in the same region or in different regions. The multiple computing devices included in the processing module 1202 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the processing module 1202 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0181] It should be noted that the information interaction, implementation process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the embodiment of the present application, and will not be repeated here.

[0182] The present application embodiment also provides a computing device 1300. Fig.13 , Fig.13 1 is a schematic diagram of a computing device 1300 provided in an embodiment of the present application. Fig.13 As shown, the computing device 1300 includes: a bus 1302, a processor 1304, a memory 1306, and a communication interface 1308. The processor 1304, the memory 1306, and the communication interface 1308 communicate through the bus 1302. The computing device 1300 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 1300.

[0183] The bus 1302 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 The bus 1302 may include a path for transmitting information between various components of the computing device 1300 (eg, the memory 1306, the processor 1304, and the communication interface 1308).

[0184] The processor 1304 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0185] The memory 1306 may include a volatile memory, such as a random access memory (RAM). The processor 1304 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0186] The memory 1306 stores executable program codes, and the processor 1304 executes the executable program codes to respectively implement the functions of the aforementioned receiving module and processing module, thereby implementing the aforementioned data processing method. That is, the memory 1306 stores instructions for executing the data processing method.

[0187] The communication interface 1308 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1300 and other devices or communication networks.

[0188] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0189] See also Fig.14 , Fig.14 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. Fig.14 As shown, the computing device cluster includes at least one computing device 1300. The memory 1306 in one or more computing devices 1300 in the computing device cluster may store the same instructions for executing the data processing method.

[0190] In some possible implementations, the memory 1306 of one or more computing devices 1300 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 1300 may jointly execute instructions for executing the data processing method.

[0191] It should be noted that the memory 1306 in different computing devices 1300 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the data processing device. That is, the instructions stored in the memory 1306 in different computing devices 1300 can implement the functions of one or more of the above-mentioned receiving module and processing module.

[0192] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.15 A possible implementation is shown. Fig.15 This is a schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application. Fig.15 As shown, in the computing device cluster 1500, two computing devices 1300A and 1300B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 1306 in the computing device 1300A stores instructions for executing the functions of the receiving module. At the same time, the memory 1306 in the computing device 1300B stores instructions for executing the functions of the processing module.

[0193] It should be understood that Fig.15 The functions of the computing device 1300A shown in FIG. 1300A may also be completed by multiple computing devices 1300. Similarly, the functions of the computing device 1300B may also be completed by multiple computing devices 1300.

[0194] See also Fig.16 , Fig.16 A schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application. The present application also provides a computer-readable storage medium. In some embodiments, the workflow executed by the above database system can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or encoded on other non-transient media or products.

[0195] Fig.16 Schematically illustrates a conceptual partial view of an example computer-readable storage medium including a computer program for executing a computer process on a computing device, arranged in accordance with at least some embodiments presented herein.

[0196] In one embodiment, the computer readable storage medium 1600 is provided using a signal bearing medium 1601. The signal bearing medium 1601 may include one or more program instructions 1602, which when executed by one or more processors may provide the functions or part of the functions described above for the above database system.

[0197] In some examples, signal bearing medium 1601 may include computer readable medium 1603 such as, but not limited to, a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, memory, ROM or RAM, and the like.

[0198] In some embodiments, the signal bearing medium 1601 may include a computer recordable medium 1604, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal bearing medium 1601 may include a communication medium 1605, such as, but not limited to, a digital and / or analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, etc.). Thus, for example, the signal bearing medium 1601 may be communicated by a wireless form of the communication medium 1605 (e.g., a wireless communication medium that complies with the IEEE 802.X standard or other transmission protocol).

[0199] The one or more program instructions 1602 may be, for example, computer executable instructions or logic implementation instructions. In some examples, the computing device of the computing device may be configured to provide various operations, functions, or actions in response to the program instructions 1602 communicated to the computing device via one or more of the computer readable medium 1603, the computer recordable medium 1604, and / or the communication medium 1605.

[0200] The present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data processing method described in the above embodiment.

[0201] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the method of each embodiment of the present application.

[0202] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

Claims

1. A data processing method, characterized in that: The method comprises: The connection processing module of the database system receives a first connection request, where the first connection request carries first user identity information of a first user; The connection processing module creates a first thread group for the first user based on the first user identity information of the first user, the first thread group is bound to the first user, and the first thread group includes at least one working thread, and the first thread group is used to process a data operation request corresponding to the first user; The connection processing module receives a second connection request, where the second connection request carries second user identity information of a second user; The connection processing module creates a second thread group for the second user based on the second user identity information of the second user, the second thread group is bound to the second user, and the second thread group includes at least one working thread, and the second thread group is used to process a data operation request corresponding to the second user; The load detection module of the database system detects the load conditions of the first thread group and the second thread group; If the load detection module detects that the duration of no load of the target thread group exceeds the target duration, the target thread group is deleted, and the target thread group is the first thread group or the second thread group.

2. The method according to claim 1, characterized in that When the connection processing module receives the first connection request, there is no thread group bound to the first user in the database system.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The connection processing module receives a third connection request, and establishes a binding relationship between a new connection established based on the third connection request and the first thread group, so that the first thread group can process a data operation request from the new connection, wherein the third connection request carries the first user identity information.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: When the load detection module detects that the average load of the thread group bound to the first user is greater than a first threshold, a new thread group bound to the first user is created.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: When the load detection module detects that the average load of the thread groups bound to the first user is less than a second threshold and the number of thread groups bound to the first user is greater than 1, some thread groups bound to the first user are deleted.

6. The method according to claim 4 or 5, characterized in that: The average load is determined based on any one or more of a plurality of load indicators, including: the number of pending data operation requests in the thread group, the number of active working threads in the thread group, and the size of processing resources occupied by the thread group.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: The load detection module creates a new working thread or deletes some of the original working threads in the first thread group based on the load condition of the first thread group.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The monitoring module of the database system obtains data operation requests from different users, and based on the users corresponding to the obtained data operation requests, transmits the obtained data operation requests to the corresponding thread group.

9. The method according to any one of claims 1 to 8, characterized in that: The first thread group further includes a request queue, which is used to store data operation requests. At least one working thread in the first thread group is used to take out the data operation request from the request queue and process the taken out data operation request.

10. The method according to any one of claims 1 to 9, characterized in that: The processing resources occupied by the first thread group and the second thread group are pre-configured.

11. The method according to any one of claims 1 to 10, characterized in that: The connection processing module and the load detection module are both implemented by one or more threads.

12. A data processing device, characterized in that: The device is applied to a database system, and comprises: A receiving module, configured to receive a first connection request, wherein the first connection request carries first user identity information of a first user; a processing module, configured to create a first thread group for the first user based on the first user identity information of the first user, wherein the first thread group is bound to the first user and includes at least one working thread, and the first thread group is configured to process a data operation request corresponding to the first user; The receiving module is further configured to receive a second connection request, where the second connection request carries second user identity information of a second user; The processing module is further used to create a second thread group for the second user based on the second user identity information of the second user, wherein the second thread group is bound to the second user and includes at least one working thread, and the second thread group is used to process a data operation request corresponding to the second user; The processing module is further used to detect load conditions of the first thread group and the second thread group; If it is detected that the duration of the target thread group without load exceeds the target duration, the processing module is further configured to delete the target thread group, where the target thread group is the first thread group or the second thread group.

13. The device according to claim 12, characterized in that When the receiving module receives the first connection request, there is no thread group bound to the first user in the database system.

14. The device according to claim 12 or 13, characterized in that The receiving module is further configured to receive a third connection request, wherein the third connection request carries the first user identity information; The processing module is further configured to establish a binding relationship between the new connection established based on the third connection request and the first thread group, so that the first thread group can process a data operation request from the new connection.

15. The device according to any one of claims 12 to 14, characterized in that: The processing module is further configured to create a new thread group bound to the first user when it is detected that the average load of the thread group bound to the first user is greater than a first threshold.

16. The device according to any one of claims 12 to 15, characterized in that: The processing module is further configured to delete some thread groups bound to the first user when it is detected that the average load of the thread groups bound to the first user is less than a second threshold and the number of thread groups bound to the first user is greater than 1.

17. The device according to claim 15 or 16, characterized in that The average load is determined based on any one or more of a plurality of load indicators, including: the number of pending data operation requests in the thread group, the number of active working threads in the thread group, and the size of processing resources occupied by the thread group.

18. The device according to any one of claims 12 to 17, characterized in that: The processing module is further configured to create a new working thread or delete some of the original working threads in the first thread group based on the load condition of the first thread group.

19. The device according to any one of claims 12 to 18, characterized in that: The processing module is further used to obtain data operation requests from different users, and based on the users corresponding to the obtained data operation requests, transmit the obtained data operation requests to the corresponding thread group.

20. The device according to any one of claims 12 to 19, characterized in that: The first thread group further includes a request queue, which is used to store data operation requests. At least one working thread in the first thread group is used to take out the data operation request from the request queue and process the taken out data operation request.

21. The device according to any one of claims 12 to 20, characterized in that: The processing resources occupied by the first thread group and the second thread group are pre-configured.

22. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-11.

23. A computer program product comprising instructions, characterized in that When the instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 11.

24. A computer storage medium, characterized in that The computer storage medium stores instructions, and when the instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 11.

Images