A database data processing method, device, system and equipment
By introducing a multi-threaded data processing mechanism into the Redis database, using thread distributors and input and output thread groups to process client requests, it solves the problem of insufficient processing capabilities of Redis in large data volume and high concurrency scenarios, and achieves more efficient data deal with.
Patent Information
- Application Number
- CN202410985993.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2025-02-07
- Estimated Expiration
- 2044-07-23
AI Technical Summary
Redis database has limited processing capabilities in large data volume and high concurrency scenarios, especially in low-latency primary and backup data replication scenarios, the existing single-threaded architecture is difficult to meet the high availability requirements.
By implementing a multi-threaded data processing mechanism on the central processor, a thread distributor is used to allocate requests to multiple service threads according to the key value requested by the client, and parsing and processing through input and output thread groups to improve data processing efficiency.
This solution significantly improves the data processing speed and capabilities of Redis database, and can more effectively utilize the advantages of CPU multi-core to handle high concurrent requests, which is suitable for large data volume scenarios.
Smart Images

Figure CN118520025B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer information technology processing technology, and in particular to a database data processing method, device, system and equipment. Background Art
[0002] Redis (an open source non-relational database) is a data structure storage that can be used as a database, cache, and message middleware. It is a Key-Value database, not a relational database. All data is stored in the server's memory in the form of Key-Value, where Value can be a variety of data structures.
[0003] The most well-known architecture is the single-threaded architecture. Although the new version starts to use multi-threading, the multi-threading mainly processes network I / O, and the read and write operations are still completed in a single thread (main thread). This results in limited processing capabilities of Redis in scenarios with large amounts of data and high concurrency. In particular, when high availability is involved, low-latency master-slave data replication is required, and Redis's existing thread mechanism is even more incapable of doing so. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a database data processing method, device, system and equipment, which can improve the data processing speed and capacity of the database.
[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:
[0006] A database data processing method, applied to a central processing unit, comprising:
[0007] Receiving a client request, wherein the client request is assigned to a service thread of a central processing unit by a thread distributor according to a key value in the client request;
[0008] Obtaining the socket corresponding to the client request, and adding the socket to a waiting queue;
[0009] According to the sockets existing in the waiting queue, the sockets are assigned to the input and output thread group for parsing and processing to obtain the target command;
[0010] Execute the target command and obtain the execution result;
[0011] The execution result is sent to the client through the input and output thread group.
[0012] Optionally, the number of service threads is equal to the number of cores of the central processing unit.
[0013] Optionally, receive client requests, including:
[0014] The receiving thread distributor performs hash calculation based on the key value of the client request and distributes the client request based on the hash calculation result.
[0015] Optionally, according to the socket existing in the waiting queue, the socket is assigned to the input and output thread group for parsing and processing to obtain the target command, including:
[0016] When the waiting queue is full, the socket connections in the waiting queue are assigned to the input and output thread group by polling;
[0017] The input and output thread group reads and parses the socket connection to obtain the target command;
[0018] Send the target command to the service thread.
[0019] Optionally, when the waiting queue is full, the socket connections in the waiting queue are assigned to the input and output thread groups by polling, including:
[0020] Create a file descriptor set based on the socket connections in the waiting queue, one file descriptor corresponds to one socket connection;
[0021] The target function is called to monitor the file descriptors in the file descriptor set. When a file descriptor is ready, the file descriptor set is traversed, and the socket connection corresponding to the file descriptor that can perform read and write operations is allocated to the input and output thread group.
[0022] Optionally, sending the execution result to the client through the input and output thread group includes:
[0023] Writing the execution result into a buffer so that the input and output thread group obtains the execution result in the buffer and writes the execution result back to the socket;
[0024] The execution result is sent to the client through the socket.
[0025] The present invention also provides a database data processing device, which is applied to a central processing unit, and the device comprises:
[0026] An acquisition module, used for receiving a client request, wherein the client request is assigned to a service thread of a central processing unit by a thread distributor according to a key value in the client request;
[0027] A processing module is used to obtain the socket corresponding to the client request and add the socket to a waiting queue; according to the socket existing in the waiting queue, the socket is assigned to an input-output thread group for parsing and processing to obtain a target command; the target command is executed to obtain an execution result; and the execution result is sent to the client through the input-output thread group.
[0028] The present invention also provides a database data processing system, comprising:
[0029] A thread distributor, for distributing the client request to a service thread of a central processing unit according to a key value of the client request;
[0030] A central processing unit, configured to receive client requests through at least two service threads, obtain a socket corresponding to the client request, assign the socket to an input / output thread group for parsing and processing, obtain a target command, and execute the target command to obtain an execution result;
[0031] At least two input and output thread groups are used to send the execution result to the client through the input and output thread groups. The present invention also provides a computing device, including: a processor, a memory storing a computer program, and when the computer program is executed by the processor, the method as described above is executed.
[0032] The present invention also provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the above method.
[0033] The above solution of the present invention includes at least the following beneficial effects:
[0034] The above scheme of the present invention receives a client request, which is assigned to the service thread of the central processing unit by the thread distributor according to the key value in the client request; obtains the socket corresponding to the client request and adds the socket to the waiting queue; according to the socket existing in the waiting queue, the socket is assigned to the input and output thread group for parsing and processing to obtain the target command; executes the target command to obtain the execution result; sends the execution result to the client through the input and output thread group; the data processing speed and capacity of the database can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 is a flowchart of a database data processing method according to an embodiment of the present invention;
[0036] Figure 2 is a thread diagram of a database data processing method according to an embodiment of the present invention;
[0037] Figure 3is a flow chart of a service thread and an input-output thread group of a database data processing method according to an embodiment of the present invention;
[0038] Figure 4 It is a structural diagram of a database data processing device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0039] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.
[0040] like Figure 1 As shown, an embodiment of the present invention provides a database data processing method, which is applied to a central processing unit, comprising:
[0041] Step 11, receiving a client request, wherein the client request is assigned to a service thread of a central processing unit by a thread distributor according to a key value in the client request;
[0042] Step 12, obtaining the socket corresponding to the client request, and adding the socket to a waiting queue;
[0043] Step 13, according to the sockets existing in the waiting queue, the sockets are assigned to the input and output thread group for parsing and processing to obtain the target command;
[0044] Step 14, executing the target command to obtain an execution result;
[0045] Step 15: Send the execution result to the client through the input and output thread group.
[0046] In this embodiment, a multi-threaded transformation mechanism is proposed to improve the data processing capability of the Redis database. Figure 2 As shown, first, multiple service threads are started according to the number of CPU cores. Each service thread can be regarded as an independent database, which processes different key values according to the customized configuration strategy. When the client initiates a command through the network connection, the client request is evenly distributed to different service threads through hash calculation according to the different key values in the request. This mechanism can greatly improve the processing capacity of Redis.
[0047] Specifically, if Figure 3As shown in the figure, when the Dispatcher (thread distributor) assigns the client request to the target service thread, the target service thread also interacts with the set IO (input and output) thread group, assigning time-consuming read and write operations to the IO (input and output) thread group for processing, while the service thread only performs command execution operations. Through the above method, Redis can make full use of the advantages of CPU multi-core and process network requests and read and write operations in parallel on different cores, thereby improving performance. It can quickly respond to more concurrent requests and is more suitable for high-concurrency and high-throughput application scenarios.
[0048] In an optional embodiment of the present invention, the number of the service threads is equal to the number of cores of the central processing unit.
[0049] In this embodiment, Figure 2 As shown, multiple TCP shared service threads (TCP shared service threads) and one IPC service thread (inter-process communication service thread) are generated, which are equivalent to the number of CPU cores. The TCP shared service thread is the above-mentioned service thread, and each service thread is bound to a CPU to give full play to the hardware performance. The IPC service thread is an independent thread. When all service threads are occupied, the IPC service thread can be used as a maintenance thread and can only be used by administrators. It is parallel to the service thread, but only serves the administrator to prevent the service thread from being occupied and unable to be managed. It can only log in locally, not remotely.
[0050] In an optional embodiment of the present invention, step 11 includes:
[0051] Step 111, the receiving thread distributor performs hash calculation according to the key value of the client request, and distributes the client request according to the hash calculation result.
[0052] In this embodiment, the thread distributor is responsible for evenly distributing the received client requests to multiple service threads. The specific distribution method is: calculate the hash value according to the key value in the client request. Then input the hash value into the remainder function to obtain the remainder of the hash value divided by the number of service threads. The remainder is matched with the label of the service thread to obtain the service thread to which the client request should be assigned. For example, assuming that the number of CPU cores is 32, 32 service threads are created, and the 32 service threads are labeled from 0 to 31 in sequence. Assume that the key value in the client request at this time is abc, and the hash value calculated based on abc is assumed to be 99. The remainder of 99 and 32 is 3, then the client request is assigned to service thread No. 3.
[0053] In an optional embodiment of the present invention, step 13 includes:
[0054] Step 131, when the waiting queue is full, the socket connections in the waiting queue are allocated to the input and output thread group by polling;
[0055] Step 132, the input and output thread group reads and parses the socket connection to obtain a target command;
[0056] Step 133, sending the target command to the service thread.
[0057] In this embodiment, Figure 3 As shown, the service thread receives the connection request from the client, obtains the socket and puts it into the waiting queue; the service thread determines that the waiting queue is full, and then assigns these connections to the IO thread group by polling; the service thread waits for the IO thread group to complete the reading of the socket; after the IO thread group completes reading the socket, the service thread starts to execute the redis command and writes the execution result to the buffer; blocks and waits for the IO thread group to write data to the socket; after waiting for the IO thread group to complete writing, the service thread clears the waiting queue, unbinds the relationship, and waits for subsequent connection requests.
[0058] In an optional embodiment of the present invention, step 131 may include:
[0059] Step 1311, creating a file descriptor set according to the socket connections existing in the waiting queue, where one file descriptor corresponds to one socket connection;
[0060] Step 1312, calling the target function to monitor the file descriptors in the file descriptor set, when a file descriptor is ready, traversing the file descriptor set, and allocating the socket connection corresponding to the file descriptor that can perform read and write operations to the input and output thread group.
[0061] In this embodiment, the file descriptor is an integer, and the operation on this integer is the operation on a file (stream). Create a socket, and a file descriptor will be returned through a system call, and the operation on the socket will be converted into the operation on the file descriptor. The target function can be a select function, which monitors the ready status of multiple IO events through the file descriptor set. The application adds the file descriptors to be monitored to the set, and then calls the select function to monitor. When a file descriptor is ready, the select function will return and inform which file descriptors are ready for reading or writing operations. The application can then process the ready IO events by traversing the file descriptor set. The socket connection corresponding to the file descriptor that can perform read and write operations is assigned to the input and output thread group.
[0062] In an optional embodiment of the present invention, step 15 may include:
[0063] Step 151, writing the execution result into a buffer, so that the input and output thread group obtains the execution result in the buffer and writes the execution result back to the socket;
[0064] The execution result is sent to the client through the socket.
[0065] In this embodiment, after the command is read, parsed, and executed, the response data of the client command is stored in the buffer, and then the service thread traverses the client queue to be written out, and assigns all tasks to the IO thread group through the RR strategy (a scheduling strategy) to write the response data back to the client. Busy polling waits for all IO thread groups to complete their tasks. Finally, the client queue to be written out is traversed again, and the command reply processor is registered for those clients that still have response data left, and the remaining response data is continued to be written back in the event loop after the client can write.
[0066] A specific embodiment of the database data processing method proposed in the embodiment of the present invention is:
[0067] When the database is started, it is started by default according to the number of CPU cores. The number of threads to be started can be set by configuring parameters. Assuming the number of threads is 32, the thread IDs are 0~31 respectively. The client sends two new requests: set key1'beijing' and set key2'shanghai'. The thread distributor receives the client request and parses the key into 'key1' and 'key2'. The hash value of the first key is 1907480701210357055 through hash calculation. The modulus of 32 is 15, and the first request is assigned to the thread with thread ID 15; the hash value of the second key is 1375100130127478633, and the modulus of 32 is 9, and the second request is assigned to the thread with thread ID 9. By analogy, all requests sent by the client can be distributed to different service threads more evenly. The service thread will check whether there is a key that is the same as the given key. If it exists, the old value will be replaced by the new value. Store key-value pairs in a data structure in memory. If the persistence option is configured, the service thread will write the data to disk asynchronously to ensure data persistence. Finally, the service thread returns a response, usually a string "OK", indicating that the operation was successful. This can greatly improve processing efficiency.
[0068] The above method of the present invention starts multiple service threads according to the number of CPU cores, each service thread processes client requests with different key values, parses the keys requested by the client, and evenly distributes them to different threads by hashing. Each service thread contains an independent replication channel. Through the setting of the above multiple service threads, client requests in large data volumes and high concurrency scenarios can be met, the performance of the hardware can be maximized, and the processing capacity of Redis can be greatly improved. The technical problem that the Redis database in the prior art can only process tasks in a single thread when executing commands and cannot process multiple threads simultaneously is solved.
[0069] like Figure 4 As shown, an embodiment of the present invention further provides a database data processing device 40, comprising:
[0070] The acquisition module 41 is used to receive a client request, where the client request is assigned to a service thread of the central processing unit by the thread distributor according to a key value in the client request;
[0071] The processing module 42 is used to obtain the socket corresponding to the client request and add the socket to a waiting queue; according to the socket existing in the waiting queue, the socket is assigned to an input-output thread group for parsing and processing to obtain a target command; the target command is executed to obtain an execution result; and the execution result is sent to the client through the input-output thread group.
[0072] Optionally, the number of service threads is equal to the number of cores of the central processing unit.
[0073] Optionally, receive client requests, including:
[0074] The receiving thread distributor performs hash calculation based on the key value of the client request and distributes the client request based on the hash calculation result.
[0075] Optionally, according to the socket existing in the waiting queue, the socket is assigned to the input and output thread group for parsing and processing to obtain the target command, including:
[0076] When the waiting queue is full, the socket connections in the waiting queue are assigned to the input and output thread group by polling;
[0077] The input and output thread group reads and parses the socket connection to obtain the target command;
[0078] Send the target command to the service thread.
[0079] Optionally, when the waiting queue is full, the socket connections in the waiting queue are assigned to the input and output thread groups by polling, including:
[0080] Create a file descriptor set based on the socket connections in the waiting queue, one file descriptor corresponds to one socket connection;
[0081] The target function is called to monitor the file descriptors in the file descriptor set. When a file descriptor is ready, the file descriptor set is traversed, and the socket connection corresponding to the file descriptor that can perform read and write operations is allocated to the input and output thread group.
[0082] Optionally, sending the execution result to the client through the input and output thread group includes:
[0083] Writing the execution result into a buffer so that the input and output thread group obtains the execution result in the buffer and writes the execution result back to the socket;
[0084] The execution result is sent to the client through the socket.
[0085] It should be noted that the device is a device corresponding to the above method, and all implementation methods in the above method embodiments are applicable to the embodiments of the device and can achieve the same technical effects.
[0086] An embodiment of the present invention further provides a database data processing system, comprising:
[0087] A thread distributor, for distributing the client request to a service thread of a central processing unit according to a key value of the client request;
[0088] A central processing unit, configured to receive client requests through at least two service threads, obtain a socket corresponding to the client request, assign the socket to an input / output thread group for parsing and processing, obtain a target command, and execute the target command to obtain an execution result;
[0089] At least two input-output thread groups are used to send the execution result to the client through the input-output thread groups.
[0090] like Figure 2 As shown, in this embodiment, the dispatcher thread is specifically responsible for allocating client requests, performing hash calculations based on the key values in the client requests, and allocating them to the corresponding service threads. By default, multiple TCP shared service threads and one IPC service thread are generated, which are equivalent to the number of CPU cores. Since the service thread can handle requests from multiple clients, it can use CPU resources most efficiently.
[0091] The embodiment of the present invention further provides a computing device, comprising: a processor, a memory storing a computer program, wherein when the computer program is executed by the processor, the method described above is executed. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.
[0092] The embodiment of the present invention further provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the above method. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.
[0093] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0094] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0095] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0096] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0097] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0098] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.
[0099] In addition, it should be pointed out that in the apparatus and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. Moreover, the steps of performing the above series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order, and some steps can be performed in parallel or independently of each other. For those of ordinary skill in the art, it is understandable that all or any steps or components of the method and apparatus of the present invention can be implemented in hardware, firmware, software or a combination thereof in any computing device (including processors, storage media, etc.) or a network of computing devices, which can be achieved by those of ordinary skill in the art using their basic programming skills after reading the description of the present invention.
[0100] Therefore, the purpose of the present invention can also be achieved by running a program or a group of programs on any computing device. The computing device can be a well-known general device. Therefore, the purpose of the present invention can also be achieved by simply providing a program product containing a program code that implements the method or device. That is to say, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any well-known storage medium or any storage medium developed in the future. It should also be pointed out that in the device and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. In addition, the steps of performing the above-mentioned series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order. Some steps can be performed in parallel or independently of each other.
[0101] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A database data processing method, characterized in that: Applicable to CPU, including: Receive a client request, wherein the client request is assigned to a service thread of a central processing unit by a thread distributor according to a key value in the client request; the number of the service threads is equal to the number of cores of the central processing unit; Obtaining the socket corresponding to the client request, and adding the socket to a waiting queue; The service thread allocates the socket to the input-output thread group for parsing according to the socket in the waiting queue to obtain the target command; the service thread interacts with the input-output thread group, the input-output thread group performs read-write operation processing, and the service thread performs execution processing of the target command; Execute the target command and obtain the execution result; Sending the execution result to the client through the input and output thread group; Receiving a client request includes: The receiving thread distributor performs hash calculation based on the key value of the client request, calculates the remainder of the hash value and the number of service threads, and distributes the client request based on the remainder; According to the sockets existing in the waiting queue, the sockets are assigned to the input and output thread groups for parsing and processing to obtain the target command, including: When the waiting queue is full, the socket connections in the waiting queue are assigned to the input and output thread group by polling; The input and output thread group reads and parses the socket connection to obtain the target command; Sending the target command to the service thread; Among them, when the waiting queue is full, the socket connections in the waiting queue are allocated to the input and output thread group by polling, including: Create a file descriptor set based on the socket connections in the waiting queue, one file descriptor corresponds to one socket connection; Call the target function to monitor the file descriptors in the file descriptor set, and when a file descriptor is ready, traverse the file descriptor set and assign the socket connection corresponding to the file descriptor that can perform read and write operations to the input and output thread group; The file descriptor is an integer, and the operation on this integer is the operation on a file; creating a socket will return a file descriptor through a system call, and the operation on the socket will be converted into the operation on the file descriptor; the target function is the system call select function, and the system call select function monitors the ready status of multiple input and output events through the file descriptor set; the application adds the file descriptors to be monitored to the set, and then the system calls the select function to monitor; when a file descriptor is ready, the system call select function will return and obtain the file descriptor that is ready for reading or writing operations, and then the application processes the ready input and output events by traversing the file descriptor set; the socket connection corresponding to the file descriptor that can perform read and write operations is allocated to the input and output thread group.
2. The database data processing method according to claim 1, characterized in that: Sending the execution result to the client through the input and output thread group includes: Writing the execution result into a buffer so that the input and output thread group obtains the execution result in the buffer and writes the execution result back to the socket; The execution result is sent to the client through the socket.
3. A database data processing device, characterized in that: Applied to a central processing unit, the device comprises: An acquisition module, used for receiving a client request, wherein the client request is assigned to a service thread of a central processing unit by a thread distributor according to a key value in the client request; the number of the service threads is equal to the number of cores of the central processing unit; A processing module is used to obtain the socket corresponding to the client request and add the socket to a waiting queue; the service thread allocates the socket to an input-output thread group for parsing and processing according to the socket in the waiting queue to obtain a target command; the service thread interacts with the input-output thread group, the input-output thread group performs read-write operation processing, and the service thread performs execution processing of the target command; executes the target command to obtain an execution result; and sends the execution result to the client through the input-output thread group; Receiving a client request includes: The receiving thread distributor performs hash calculation based on the key value of the client request, calculates the remainder of the hash value and the number of service threads, and distributes the client request based on the remainder; According to the sockets existing in the waiting queue, the sockets are assigned to the input and output thread groups for parsing and processing to obtain the target command, including: When the waiting queue is full, the socket connections in the waiting queue are assigned to the input and output thread group by polling; The input and output thread group reads and parses the socket connection to obtain the target command; Sending the target command to the service thread; Among them, when the waiting queue is full, the socket connections in the waiting queue are allocated to the input and output thread group by polling, including: Create a file descriptor set based on the socket connections in the waiting queue, one file descriptor corresponds to one socket connection; Call the target function to monitor the file descriptors in the file descriptor set, and when a file descriptor is ready, traverse the file descriptor set and assign the socket connection corresponding to the file descriptor that can perform read and write operations to the input and output thread group; The file descriptor is an integer, and the operation on this integer is the operation on a file; creating a socket will return a file descriptor through a system call, and the operation on the socket will be converted into the operation on the file descriptor; the target function is the system call select function, and the system call select function monitors the ready status of multiple input and output events through the file descriptor set; the application adds the file descriptors to be monitored to the set, and then the system calls the select function to monitor; when a file descriptor is ready, the system call select function will return and obtain the file descriptor that is ready for reading or writing operations, and then the application processes the ready input and output events by traversing the file descriptor set; the socket connection corresponding to the file descriptor that can perform read and write operations is allocated to the input and output thread group.
4. A database data processing system, characterized in that: include: A thread distributor, for distributing the client request to a service thread of a central processing unit according to a key value of the client request; The number of service threads is equal to the number of cores of the central processing unit; A central processing unit, configured to receive client requests through at least two service threads, obtain a socket corresponding to the client request, and add the socket to a waiting queue; The service thread allocates the socket to the input and output thread group according to the socket in the waiting queue for parsing and processing to obtain the target command; The service thread interacts with the input-output thread group, the input-output thread group performs read-write operation processing, and the service thread performs execution processing of the target command; executes the target command to obtain an execution result; at least two input-output thread groups, used to send the execution result to the client through the input-output thread groups; Receiving a client request includes: The receiving thread distributor performs hash calculation based on the key value of the client request, calculates the remainder of the hash value and the number of service threads, and distributes the client request based on the remainder; According to the sockets existing in the waiting queue, the sockets are assigned to the input and output thread groups for parsing and processing to obtain the target command, including: When the waiting queue is full, the socket connections in the waiting queue are assigned to the input and output thread group by polling; The input and output thread group reads and parses the socket connection to obtain the target command; Sending the target command to the service thread; Among them, when the waiting queue is full, the socket connections in the waiting queue are allocated to the input and output thread group by polling, including: Create a file descriptor set based on the socket connections in the waiting queue, one file descriptor corresponds to one socket connection; Call the target function to monitor the file descriptors in the file descriptor set, and when a file descriptor is ready, traverse the file descriptor set and assign the socket connection corresponding to the file descriptor that can perform read and write operations to the input and output thread group; The file descriptor is an integer, and the operation on this integer is the operation on a file; creating a socket will return a file descriptor through a system call, and the operation on the socket will be converted into the operation on the file descriptor; the target function is the system call select function, and the system call select function monitors the ready status of multiple input and output events through the file descriptor set; the application adds the file descriptors to be monitored to the set, and then the system calls the select function to monitor; when a file descriptor is ready, the system call select function will return and obtain the file descriptor that is ready for reading or writing operations, and then the application processes the ready input and output events by traversing the file descriptor set; the socket connection corresponding to the file descriptor that can perform read and write operations is allocated to the input and output thread group.
5. A computing device, characterized in that: include: A processor and a memory storing a computer program, wherein when the computer program is executed by the processor, the method according to any one of claims 1 to 2 is performed.
6. A computer-readable storage medium, characterized in that: Instructions are stored, and when the instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 2.
Citation Information
Patent Citations
Memory-oriented multi-thread database design method
CN107590199A
Redis-based data processing method and apparatus, and server device
CN112202766A