Data processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of data processing, in particular to a data processing method and device, electronic equipment and a storage medium, which are used for preventing a working thread of a software program initiating IO operation from being blocked, saving processing resources and improving the processing efficiency of the IO operation. The method comprises the steps that when IO operation is executed every time, a processing function corresponding to the IO operation is created, an operation task corresponding to the IO operation is added to a to-be-executed task queue, and the processing function is used for obtaining a corresponding operation result after the operation task is executed; in the process of executing each operation task in the to-be-executed task queue, querying the processing state of each operation task in the to-be-executed task queue; when it is queried that the processing state of the target operation task is completed, the target operation task is added to an executed task queue; and for each target operation task in the executed task queue, executing a processing function corresponding to the target operation task to obtain a corresponding operation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a data processing method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In an electronic device (such as a terminal device or a server), a software program usually needs to perform input / output (IO) operations through an operating system, and the IO operations include data reading and writing, network communication, control of a hardware device, and the like. For example, taking the electronic device as a server and the IO operation as data reading and writing as an example, when the server receives a data acquisition request of a terminal device, a database software (i.e., a software program) in the server calls the operating system to read corresponding data from a hard disk, so as to send the read data to the terminal device.

[0003] In some scenarios, a software program of an electronic device may need to perform a large number of IO operations through an operating system. However, in the related art, when a work thread of the software program initiates an IO operation to the operating system, the work thread needs to wait for the completion of the execution of the IO operation, and then perform other tasks of the software program, for example, initiates a new IO operation to the operating system. Since the execution of each IO operation needs a certain time, the work thread of the software program is blocked, processing resources are wasted, and the processing efficiency is low when a large number of IO operations need to be processed. SUMMARY

[0004] Embodiments of the present application provide a data processing method and device, electronic equipment and a storage medium, to avoid the work thread of a software program initiating an IO operation from being blocked, save processing resources, and improve the processing efficiency of the IO operation.

[0005] In an aspect, the present application provides a data processing method, which comprises:

[0006] Each time an input / output (IO) operation is performed, a processing function corresponding to the IO operation is created, and an operation task corresponding to the IO operation is added to a to-be-executed task queue; wherein the processing function is used to obtain an operation result corresponding to the operation task after the operation task is executed;

[0007] In the process of executing each operation task in the to-be-executed task queue, the processing state of each operation task in the to-be-executed task queue is queried;

[0008] When the processing state of a target operation task in the to-be-executed task queue is queried to be completed, the target operation task is added to an executed task queue;

[0009] For at least one target operation task in the executed task queue, the following operations are respectively performed: a processing function corresponding to one target operation task is executed to obtain an operation result of the one target operation task.

[0010] In one aspect, the data processing apparatus provided by the embodiments of the present application comprises:

[0011] The creating unit is configured to create a processing function corresponding to an input / output (IO) operation and add an operation task corresponding to the IO operation to a to-be-executed task queue each time the IO operation is performed, wherein the processing function is used to obtain a corresponding operation result after the operation task is executed.

[0012] The querying unit is configured to query a processing state of each operation task in the to-be-executed task queue in a process of executing each operation task in the to-be-executed task queue.

[0013] The transferring unit is configured to add a target operation task in the to-be-executed task queue to an executed task queue when it is queried that the processing state of the target operation task is completed.

[0014] The processing unit is configured to, for at least one target operation task in the executed task queue, respectively perform the following operations: a processing function corresponding to one target operation task is executed to obtain an operation result of the one target operation task.

[0015] Optionally, the apparatus further comprises:

[0016] The registering unit is configured to register a descriptor corresponding to the operation task to an input / output (IO) multiplexing interface of the operating system, wherein the IO multiplexing interface is used to detect a state of at least one descriptor, so that the operating system executes a corresponding operation task according to the state of each descriptor to obtain a processing state of the operation task.

[0017] The querying unit is specifically configured to:

[0018] In a process of executing each operation task in the to-be-executed task queue by the operating system, each operation task in the to-be-executed task queue is sequentially traversed every set period, wherein each time an operation task is traversed, a processing state of the corresponding operation task is queried from the IO multiplexing interface based on a descriptor corresponding to the one operation task.

[0019] Optionally, the apparatus further comprises:

[0020] The waking unit is configured to send task wake-up information for any operation task in the to-be-executed task queue to the operating system when it is determined that the any operation task is in an uncompleted state within a set time length.

[0021] The task wake-up information is used to indicate processing of the operation task.

[0022] Optionally, the transferring unit is specifically used for:

[0023] When it is queried that the processing state of a target operation task in the to-be-executed task queue is completed, the target operation task is deleted from the to-be-executed task queue;

[0024] The target operation task and the processing function corresponding to the target operation task are added to the executed task queue.

[0025] Optionally, the plurality of target operation tasks in the executed task queue are arranged in a completed order;

[0026] The processing unit is specifically used for:

[0027] When it is determined that a work thread for processing a target operation task is idle, a processing function corresponding to each target operation task is obtained from the executed task queue in a sequence based on an arrangement order of at least one target operation task in the executed task queue, and the obtained processing function is executed to obtain an operation result of the target operation task.

[0028] Optionally, the apparatus further comprises:

[0029] The merging unit is used for merging at least two operation tasks in the to-be-executed operation queue when the at least two operation tasks satisfy a set merging condition, and obtaining a merged operation task.

[0030] The set merging condition comprises at least one of the following:

[0031] IO operation types are the same;

[0032] IO resources are the same.

[0033] Optionally, the apparatus further comprises:

[0034] The setting unit is used for setting priority information for each operation task in the to-be-executed task queue according to a set priority rule, and sorting the operation tasks according to the priority information.

[0035] The querying unit is specifically used for:

[0036] In a process of executing each operation task in the to-be-executed task queue, a processing state of each operation task is queried in a sequence according to a priority arrangement order of the operation tasks.

[0037] In an aspect, an electronic device is provided, which includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the steps of any of the above data processing methods.

[0038] In an aspect, a computer-readable storage medium is provided, which includes a computer program. When the computer program is run on an electronic device, the computer program is configured to cause the electronic device to perform the steps of any of the above data processing methods.

[0039] In an aspect, a computer program product is provided, which includes a computer program stored in a computer-readable storage medium. When a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any of the above data processing methods.

[0040] The scheme of the present application has at least the following beneficial effects:

[0041] The embodiments of the present application provide a data processing method and device, an electronic device and a storage medium. Each time an IO operation is performed, a processing function of the IO operation is created, which is used to obtain a processing result of an operation task corresponding to the IO operation. Thus, the IO operation does not need to be waited for completion, and other tasks, such as a new IO operation or other operations, can be performed. In addition, each operation task corresponding to an IO operation is added to a to-be-executed task queue, so that a plurality of operation tasks in the to-be-executed task queue are conveniently scheduled. A completed target operation task in the plurality of operation tasks is added to an executed task queue, so that the processing function of the completed target operation task is sequentially called, and the processing result of the target operation task is obtained. Therefore, when an IO operation is initiated, the working thread of a software program can be prevented from being blocked, processing resources are saved, and when a large number of IO operations need to be processed, the processing efficiency of the IO operations is improved.

[0042] Other features and advantages of the present application will be further described in the following specification, and will be apparent to one of ordinary skill in the art. The purposes and other advantages of the present application can be realized and achieved by the structure particularly pointed out in the written specification, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the specification, illustrate the illustrative embodiments of the present application and serve to explain the present application, and do not limit the present application in any manner. In the drawings:

[0044] Figure 1 An application scenario of a data processing method provided by an embodiment of the present application is shown in the figure;

[0045] Figure 2 An application scenario of a data processing method provided by an embodiment of the present application is shown in the figure;

[0046] Figure 3 An application scenario of a data processing method provided by an embodiment of the present application is shown in the figure;

[0047] Figure 4 An application scenario of a data processing method provided by an embodiment of the present application is shown in the figure;

[0048] Figure 5 An application scenario of a data processing method provided by an embodiment of the present application is shown in the figure;

[0049] Figure 6 An application scenario of a data processing method provided by an embodiment of the present application is shown in the figure;

[0050] Figure 7 An application scenario of a data processing method provided by an embodiment of the present application is shown in the figure;

[0051] Figure 8 An application scenario of a data processing method provided by an embodiment of the present application is shown in the figure;

[0052] Figure 9 An application scenario of a data processing method provided by an embodiment of the present application is shown in the figure;

[0053] Figure 10 An application scenario of a data processing method provided by an embodiment of the present application is shown in the figure;

[0054] Figure 11 An application scenario of a data processing method provided by an embodiment of the present application is shown in the figure;

[0055] Figure 12 An application scenario of a data processing method provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments described in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0057] Some concepts involved in the embodiments of the present application are introduced below.

[0058] IO operation: is a basic activity in computer system, involving the transmission of data between computer internal and external devices. In computer science, IO operation usually refers to the exchange of data between software programs or operating systems and external devices, which can be physical hardware such as disk drives, keyboards, displays, network interfaces, etc., or software-level concepts such as files, pipes, sockets, etc. IO operation can be divided into read operation and write operation; read operation is to read data from external device to computer memory, for example, read file content from disk, receive data packet from network; write operation is to write data from computer memory to external device; for example, write data to disk as a file, send data packet to network.

[0059] Rust: is a general-purpose, compiled programming language. The design criteria are "safety, concurrency, utility", supporting functional, concurrent, procedural and object-oriented programming styles.

[0060] Asynchronous programming: is a programming mode that allows the program to continue executing other code while waiting for some operations (such as IO operations, etc.) to complete, rather than blocking. When these operations are completed, an asynchronous event will be triggered, and the corresponding callback function or asynchronous handler will be executed.

[0061] Asynchronous event: Asynchronous event refers to the event that does not follow the order of code execution, but is executed when certain conditions are met or a specific time point is reached. The processing of asynchronous events usually depends on callback functions, Promises, or asynchronous / await mechanisms. Among them, Promises, asynchronous / await mechanisms belong to the writing mode of asynchronous handlers.

[0062] Callback function: is a custom function that is passed as a parameter to another function (usually called caller or higher-order function), and is called by another function under certain conditions. This mechanism allows registering a piece of code to respond to future events or conditions without determining when or whether they will occur.

[0063] Epoll: is an IO multiplexing technology provided by Linux operating system. IO multiplexing is a technology that handles multiple IO events in a single thread, which can improve the efficiency of program processing IO events.

[0064] The word "exemplary" used in the following means "as an example, embodiment or illustrative". Any embodiment described as "exemplary" does not necessarily mean that it is superior or better than other embodiments.

[0065] The design principles of the embodiments of this application will be introduced below.

[0066] In electronic devices (such as terminal devices or servers), software programs typically need to perform input / output (I / O) operations through the operating system. In some scenarios, the software programs of electronic devices may need to perform a large number of I / O operations through the operating system; for example, server software programs need to handle a large number of network requests from clients to perform corresponding data sending and receiving operations (i.e., I / O operations). However, in related technologies, when a software program's worker thread initiates an I / O operation to the operating system, it needs to wait for that I / O operation to complete before executing other tasks of the software program, such as initiating new I / O operations to the operating system. Since the execution of each I / O operation takes a certain amount of time, this can cause the software program's worker thread to be blocked, wasting processing resources, and resulting in low processing efficiency when a large number of I / O operations need to be processed.

[0067] In view of this, embodiments of this application provide a data processing method, apparatus, electronic device, and storage medium. For each I / O operation executed, a processing function for that I / O operation is created for subsequently obtaining the processing result of the corresponding operation task. This eliminates the need to wait for the I / O operation to complete before executing other tasks, such as initiating new I / O operations or other operations. Furthermore, the operation task corresponding to each I / O operation is added to a queue of tasks to be executed, facilitating the scheduling of multiple operation tasks in the queue. Completed target operation tasks are transferred to an executed task queue, allowing for the sequential invocation of the processing functions of completed target operation tasks to obtain their processing results. Therefore, embodiments of this application can avoid blocking the working threads of software programs when initiating I / O operations, saving processing resources, and improving the processing efficiency of I / O operations when handling a large number of I / O operations.

[0068] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0069] like Figure 1 The diagram shown is an application scenario illustration of an embodiment of this application. The application scenario diagram includes a terminal device 110 and a server 120.

[0070] In one alternative implementation, the terminal device 110 and the server 120 can communicate via a communication network; wherein the communication network is a wired network or a wireless network.

[0071] In this embodiment, the terminal device 110 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, and in-vehicle terminals. The terminal device may have a client installed, which can be software (e.g., browsers, video software, shopping software, music software), or a webpage, mini-program, etc. The server 120 can be a backend server corresponding to the software, webpage, or mini-program, or a server specifically used for data processing; this application does not impose specific limitations. The server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0072] It should be noted that the data processing methods in the various embodiments of this application can be executed by an electronic device, which can be a terminal device 110 or a server 120. That is, the method can be executed by either the terminal device 110 or the server 120. The following description uses server execution as an example.

[0073] The software program in server 120 can initiate I / O operations to the operating system. These I / O operations include file I / O operations (i.e., data reading operations) and network I / O operations (i.e., data sending and receiving operations). File I / O operations require the operating system to interact with storage devices (such as disks, hard drives, etc.) to read data from the storage device into memory or write data in memory to the storage device. Network I / O operations require the operating system to receive and send network data.

[0074] For file I / O operations, such as Figure 2 As shown, when a software program reads data from a storage device, it sends a data read request to the operating system. The operating system reads the data from the storage device into the kernel buffer in the kernel space, and then copies the data in the kernel buffer to the user space memory for the software program to use. Similarly, when a software program writes data to a storage device, it sends a data write request to the operating system. The operating system copies the data in the user space memory to the kernel buffer in the kernel space, and then writes the data in the kernel buffer to the storage device.

[0075] For network I / O operations, such as Figure 3As shown, when a software program receives network data, such as network data sent by a client in terminal device 110, the software program sends a data receiving request to the operating system, the operating system receives network data from a network device (such as a network interface, etc.) to a kernel buffer in the kernel space, and then copies the network data in the kernel buffer to the memory in the user space for the software program in server 120 to read; similarly, when a software program sends network data, such as network data sent to a client in terminal device 110, the software program sends a data sending request to the operating system, the operating system copies the network data in the memory in the user space to the kernel buffer in the kernel space, and then sends the network data through the network device.

[0076] In some scenarios, taking the server 120 performing network IO operations as an example, in the process of communication between server 120 and terminal device 110, the software program in server 120 needs to initiate an IO operation to the operating system to receive network data sent by a client in terminal device 110, or to send network data to a client in terminal device 110. In addition, server 120 can also initiate an IO operation to the operating system through a software program in the process of network communication with other devices (other terminal devices or servers), and this is not limited.

[0077] For example, server 120 is a Web (World Wide Web, global wide area network) server, when the Web server receives an HTTP (HyperText Transfer Protocol, HyperText Transfer Protocol) request from a client (such as a browser), it reads the request data through a network IO operation, then processes the request data (such as finding a file, running a script, querying a database, etc.), and finally sends response data back to the client through a network IO operation.

[0078] For another example, server 120 is a database server, when the database server receives a SQL (Structured Query Language, Structured Query Language) query request from a client through a network IO operation, processes the query request (such as executing a SQL statement, retrieving data, etc.), and then returns the query result to the client through a network IO operation.

[0079] In other scenarios, taking the server 120 performing file IO operations as an example, when a software program in server 120 reads data in a storage device or writes data to a storage device, it can initiate an IO operation to the operating system.

[0080] For example, the server is a web server, which records access logs including time of request, source IP (Internet Protocol) address, requested URL (Uniform Resource Locator), response status code, and the like. These access logs are usually written into log files for subsequent analysis. After each HTTP request is processed, the web server can write the access log into a log file in the storage device through file IO operation.

[0081] For another example, the server 120 is a database server, which frequently performs file IO operations when processing data. For example, when adding data to the database and updating existing data, the data is written into a database file in the storage device, or when retrieving data from the database, the database file is read from the storage device.

[0082] For another example, in backup and recovery operations, the server 120 needs to perform a large number of file IO operations to read and write data. For example, when performing data backup, the server traverses the specified directory, reads the file and copies it to the backup medium (such as a tape, an external hard disk); when performing data recovery, the server reads the file from the backup medium and writes it back to the original location.

[0083] In the above scenarios, when the software program in the server 120 initiates an IO operation to the operating system, the data processing method of the embodiments of the present application can be used, so that the software program does not need to wait for the completion of the IO operation when initiating the IO operation, that is, it can perform other tasks, such as initiating a new IO operation or other operations, avoiding the blocking of the working thread of the software program, saving processing resources; and adding the operation task corresponding to the IO operation to the to-be-executed task queue, thereby facilitating the scheduling of multiple operation tasks in the to-be-executed task queue, transferring the completed target operation task in the multiple operation tasks to the executed task queue, thereby facilitating the calling of the processing function of the completed target operation task in sequence, obtaining the processing result of the target operation task, and being beneficial to processing a large number of IO operations and improving the processing efficiency of the IO operations.

[0084] It should be noted that the process of initiating an IO operation by the software program (such as a client) in the terminal device to the operating system is similar to the process of initiating an IO operation by the software program in the server to the operating system, which will not be described here.

[0085] Figure 1 It is only an example, and in fact the number of terminal devices and servers is not limited, which is not specifically limited in the embodiments of the present application.

[0086] The data processing method provided by the exemplary embodiments of the present application is described below in combination with the above-described application scenarios and with reference to the accompanying drawings. It should be noted that the above-described application scenarios are only shown for the purpose of facilitating the understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect.

[0087] Referring to Figure 4 FIG. 4 shows an implementation flowchart of the data processing method provided by the embodiments of the present application, taken the server as an execution subject, the specific implementation flow of the method includes the following S41-S44:

[0088] S41, each time an IO operation is performed, a processing function corresponding to the IO operation is created, and an operation task corresponding to the IO operation is added to a task queue to be executed; wherein the processing function is used to obtain an operation result corresponding to the operation task after the operation task is executed.

[0089] In the server, a software program (a program providing various services) needs to perform an IO operation through an operating system, for example, the file IO operation or the network IO operation in the above-described embodiments. After the operation is performed, the operating system can deliver an operation result, such as read data, a data write state (such as the number of written bytes), received network data, or a network data sending state, to the software program.

[0090] In the embodiments of the present application, in order not to block the working thread of the software program and to enable the working thread to continue to execute other tasks during the execution of the IO operation by the operating system, each time an IO operation is performed, a processing function corresponding to the IO operation is created, and the processing function is registered with the operating system. After the IO operation is performed, the operating system can call a callback function to deliver an operation result of the IO operation to a receiving parameter of the callback function.

[0091] Optionally, the processing function can be a pre-defined callback function, and the callback function defines a data type of a receiving parameter and a data type of a return parameter. The data type of the receiving parameter indicates a type of a value that can be delivered to the receiving parameter, such as an integer, a floating point number, a string, etc. The data type of the return parameter indicates a type of a value returned after the callback function is executed, such as an integer, a floating point number, a string, etc. After the IO operation is performed, the operating system can call the callback function, i.e., execute the code logic defined in the callback function, to deliver the operation result of the IO operation to the receiving parameter of the callback function. Then, the software program can obtain the operation result of the IO operation by executing the callback function. It should be noted that the data type of the receiving parameter of the callback function is consistent with the data type of the operation result of the operation task.

[0092] Further, in order to facilitate the execution of multiple IO operations, an operation task corresponding to each IO operation is created, and each operation task corresponding to each IO operation is added to the to-be-executed task queue, so that the completion progress of multiple operation tasks is detected in sequence. Specifically, the to-be-executed task queue can be in the form of a list, or in other forms, which are not limited.

[0093] As shown in the example, Figure 5 The to-be-executed task queue includes operation task 1, operation task 2, operation task 3,..., and operation task n. When a new IO operation is initiated, a new operation task corresponding to the new IO operation is created, the new operation task is added to the to-be-executed task queue, and operation task n+1 is obtained.

[0094] In some embodiments, when the operation task corresponding to the IO operation is added to the to-be-executed task queue in S201, the operation task corresponding to the IO operation can be created, and an IO operation request corresponding to the IO operation can be initiated to the operating system; then, a descriptor of an IO resource required for executing the IO operation is obtained from the operating system, and a processing function corresponding to the IO operation is registered with the operating system; finally, the operation task carrying the descriptor is added to the to-be-executed task queue; wherein the descriptor is used to identify the IO resource.

[0095] Specifically, an operation task is created for the to-be-executed IO operation, and the operation task is an asynchronous task, that is, the software program does not need to wait for the execution of the operation task, and can continue to execute other tasks. The software program can initiate an IO operation request corresponding to the operation task to the operating system by calling a specified system call of the operating system. For example, when the IO operation is a file IO operation, a file can be opened, and the operating system allocates a file descriptor for the file. If it is necessary to read data in the file or write data into the file, an IO operation request can be initiated to the operating system; when the IO operation is a network IO operation, a network connection can be opened, and the operating system allocates a network descriptor for a network socket (i.e., an endpoint of network communication) of the network connection. If it is necessary to receive network data or send network data, an IO operation request can be initiated to the operating system. It can be understood that the IO resource can be a file, a network socket, etc., and the corresponding descriptor can be a file descriptor or a network descriptor.

[0096] The descriptor is a unique identifier of the opened file or network socket, and the descriptor can enable the software program to communicate with the operating system for the IO operation in the IO resource. Thus, the processing status of the operation task corresponding to the IO operation can be obtained through the descriptor, and therefore, the operation task carrying the descriptor is added to the to-be-executed task queue.

[0097] In the embodiments of this application, after an IO operation request corresponding to the IO operation initiated to the operating system is obtained, a descriptor of an IO resource required for performing the IO operation is obtained, and an operation task carrying the descriptor is added to a to-be-executed task queue, so that the processing state of the operation task can be queried subsequently.

[0098] In some embodiments, after the operation task carrying the descriptor is added to the to-be-executed task queue, the descriptor corresponding to the operation task can be registered to an IO multiplexing interface of the operating system; wherein the IO multiplexing interface is used to detect the state of at least one descriptor, so that the operating system can execute a corresponding operation task according to the state of each descriptor to obtain the processing state of the operation task.

[0099] The IO multiplexing is a technology for detecting the state of multiple descriptors (such as file descriptors, network descriptors, etc.) in a single process at the same time, which enables the working thread of a software program to handle multiple IO operations at the same time. When the IO multiplexing interface of the operating system detects that the state of a descriptor is readable or writable, it indicates that the IO operation on the file or network socket identified by the descriptor is ready, and the IO operation can be started. At this time, the operating system kernel can execute the IO operation (i.e., the operation task), and the IO multiplexing interface can also detect the processing state of the IO operation corresponding to each descriptor; wherein the file can correspond to the file descriptor, and the network socket can correspond to the network descriptor.

[0100] Different operating systems can correspond to different IO multiplexing interfaces, for example, the IO multiplexing interface in Unix or Linux operating system is epoll (Event Poll, event polling), the IO multiplexing interface in Windows operating system is kqueue (Kernel Queue, event queue in kernel), and the IO multiplexing interface in macOS operating system is IOCP (Input / Output Completion Ports, Input / Output Completion Ports).

[0101] The process of detecting the state of the descriptor by the IO multiplexing interface is described below.

[0102] For example, the data processing method of the embodiments of this application can be executed by an asynchronous event-driven engine provided in the server. The asynchronous event-driven engine can be programmed asynchronously based on the Rust programming language, allowing a software program to perform an IO operation without blocking a working thread, i.e., implementing non-blocking asynchronous IO operation, which is achieved by integrating with the IO multiplexing interface (such as epoll, kqueue, IOCP, etc.) of the operating system.

[0103] Below, take the epoll interface in the Linux operating system as an example, and exemplarily introduce the way of using the epoll interface in Rust to realize asynchronous events.

[0104] Firstly, add the third-party library nix as a dependency in the Cargo.toml file. Wherein, nix refers to a Rust programming language library that provides a safe wrapper around the Linux operating system's APIs, allowing Rust programmers to access Unix system's underlying functionality such as file operations in a type-safe and easy-to-use manner. The Cargo.toml file is a core configuration file for Rust projects, used to describe the metadata and dependency relationships of Rust projects. Then, based on the nix library, the epoll interface of the operating system is called to realize asynchronous events. The epoll interface is used to efficiently detect the state of a large number of file descriptors.

[0105] Next, create a Rust project nix crate, and create a new epoll instance through the epoll_create1 function in the nix crate. Wherein, crate is the basic unit of a Rust project, and a crate can be understood as a separate compilation unit, which can be an executable program or a library.

[0106] Then, open a file and set the file to non-blocking mode, and the operating system assigns a file descriptor to the file, so that the state of the file descriptor of the file can be detected through the epoll interface. Specifically, create an EpollEvent structure, which indicates interest in reading events for the file descriptor; call the epoll_ctl system call to add the file descriptor to the interest list of the epoll instance, and then call the epoll_wait system call to wait for asynchronous events to occur. The epoll_wait call will block until at least one asynchronous event occurs; iterate through the asynchronous events that have occurred, check if the asynchronous event is related to the detected file descriptor, and if so, handle the asynchronous event. Wherein, the asynchronous event refers to the state of the file descriptor being readable or writable, and the read-write operation can be performed on the file identified by the file descriptor.

[0107] Finally, close the file descriptor and the epoll file descriptor to clean up resources. Wherein, the epoll file descriptor represents the above-mentioned epoll instance.

[0108] In the embodiments of the present application, the state of the descriptor corresponding to each operation task can be detected through the IO multiplexing interface of the operating system, so that the operating system executes the corresponding operation task according to the state of each descriptor, and obtains the processing state of the operation task. For example, when the state of the descriptor is readable or writable, the operating system executes the data read / write (or transmits / receives) operation on the file (or network socket) identified by the descriptor. In this way, the software program can process multiple IO operations (i.e., operation tasks) at the same time.

[0109] S42, in the process of executing each operation task in the to-be-executed task queue, querying the processing state of each operation task in the to-be-executed task queue.

[0110] In some embodiments, when the operation task carrying the descriptor is added to the to-be-executed task queue, in the process of querying the processing state of each operation task in turn, the processing state of the corresponding operation task can be queried from the IO multiplexing interface based on the descriptor corresponding to each operation task.

[0111] In some embodiments, when the operation task carrying the descriptor is added to the to-be-executed task queue, in the process of querying the processing state of each operation task in turn, the processing state of the corresponding operation task can be queried from the IO multiplexing interface based on the descriptor corresponding to each operation task.

[0112] Further, each operation task in the to-be-executed task queue can be traversed in turn every set period, and the processing state of the corresponding operation task can be queried from the IO multiplexing interface based on the descriptor corresponding to each operation task every time an operation task is traversed.

[0113] In some embodiments, the set period can be set as needed, for example, several seconds, which is not limited. Specifically, the to-be-executed task queue is continuously polled (i.e., queried every set period) until the operation tasks in the to-be-executed task queue are executed. Since the descriptor corresponding to each operation task is registered in the IO multiplexing interface in advance, and the state of each descriptor (such as readable or writable) is detected by the IO multiplexing interface, so that the operating system kernel executes the corresponding operation task according to the state of each descriptor, and obtains the processing state of the operation task, therefore, in each query, the processing state of the corresponding operation task can be queried from the IO multiplexing interface based on the descriptor corresponding to the operation task.

[0114] In the embodiments of the present application, the processing status of each operation task is queried by polling each operation task in the to-be-executed task queue, so as to facilitate scheduling of each operation task. Specifically, the descriptor corresponding to each operation task can facilitate querying from the operating system whether the operation task is completed, thereby improving the querying efficiency.

[0115] Based on the above-mentioned embodiments of the present application, the data processing method of the embodiments of the present application can be executed by using a Rust-based asynchronous event-driven engine. The asynchronous event-driven engine includes a task scheduler, and each operation task in the to-be-executed task queue is scheduled by the task scheduler. The IO operation corresponding to each operation task is an asynchronous operation. The following exemplary introduces how to implement a task scheduler in Rust.

[0116] Implementing a task scheduler in Rust usually involves asynchronous programming. The standard library of Rust provides a Future feature, which is the basis of asynchronous operations and is used to express a calculation that may be completed at a future time. Specifically, a task scheduler Scheduler is defined, and some asynchronous operation tasks are run by the task scheduler Scheduler. In the task scheduler, a Task structure is created to encapsulate the Future feature, which is used to represent an operation task, and a to-be-executed task queue is managed in the task scheduler Scheduler. The task scheduler polls each operation task in the to-be-executed task queue by a polling method (such as a run method). If an operation task has not been completed, the operation task is put back into the to-be-executed task queue. The task scheduler will continue to run until all operation tasks are completed.

[0117] The Task structure can also include status information of the operation task, such as the ID and priority of the operation task, which helps to better schedule and identify the operation task in the task scheduler.

[0118] S43、In the query to the processing status of the target operation task in the to-be-executed task queue is completed, the target operation task is added to the executed task queue.

[0119] Each time a completed target operation task is queried, the current queried target operation task cannot be processed immediately due to some reasons (such as being executed by other tasks). Therefore, the target operation task is added to the executed task queue. Each target operation task in the executed task queue can be arranged in the order of completion, so as to facilitate subsequent execution of each target operation task in the executed task queue in turn.

[0120] In some embodiments, when it is found that the processing state of a target operation task in the to-be-executed task queue is completed, the target operation task is deleted from the to-be-executed task queue; and the target operation task and the processing function corresponding to the target operation task are added to the executed task queue.

[0121] When the target operation task is executed, the operation system can deliver the operation result of the target operation task to the corresponding processing function. In order to facilitate the execution of the processing function, when the target operation task is added to the executed task queue, the processing function corresponding to the target operation task can also be added to the executed task queue.

[0122] For example, as shown in FIG. 1, the to-be-executed task queue includes operation task 1, operation task 2, operation task 3, …, operation task n+1. When it is found that the processing state of operation task 3 is completed, operation task 3 is added to the executed task queue. The executed task queue already includes target operation task 1, target operation task 2, target operation task 3, …, target operation task m. Operation task 3 is taken as target operation task m+1. Meanwhile, each target operation task can also carry a corresponding processing function. Figure 6

[0123] In the embodiments of the present application, after the target operation task that has been executed is deleted from the to-be-executed task queue, the target operation task and the processing function corresponding to the target operation task are added to the executed task queue. Thus, it is convenient to subsequently obtain the processing function of the target operation task in turn and execute the processing function to obtain the processing result of the target operation task.

[0124] S44, for at least one target operation task in the executed task queue, the following operations are performed respectively: executing the processing function corresponding to one target operation task to obtain the operation result of one target operation task.

[0125] When the executed task queue includes multiple target operation tasks, the multiple target operation tasks can be arranged in the order of completion, i.e., the target operation task that is completed first is arranged in front and is executed first. The executed task queue contains each target operation task and the corresponding processing function. Thus, for each target operation task, the processing function of the target operation task can be obtained from the executed task queue for execution.

[0126] ​In the embodiments of the present application, a processing function of an IO operation is created each time the IO operation is initiated, which is used to obtain the processing result of the operation task corresponding to the IO operation subsequently. In this way, the other tasks, such as initiating a new IO operation or other operations, can be executed without waiting for the completion of the IO operation. In addition, the operation task corresponding to each IO operation is added to the to-be-executed task queue, thereby facilitating the scheduling of the multiple operation tasks in the to-be-executed task queue and the transfer of the completed target operation task in the multiple operation tasks to the executed task queue, so as to facilitate the calling of the processing function of the completed target operation task in sequence and the obtaining of the processing result of the target operation task. Therefore, the embodiments of the present application can avoid the blocking of the working thread of the software program when initiating the IO operation, save the processing resources, and improve the processing efficiency of the IO operation when a large number of IO operations need to be processed.

[0127] In some embodiments, the working thread of the software program can continue to execute other tasks after initiating the IO operation each time. When the target operation task exists in the executed task queue after the execution of one or more IO operations, in order not to affect the execution of the other tasks, the one or more target operation tasks in the executed task queue can be executed again when it is determined that the working thread of the software program is idle.

[0128] Therefore, in S44 of the above embodiments of the present application, when the target operation task in the executed task queue is processed, it can be determined whether the working thread used to process the target operation task is idle. If it is idle, the processing function corresponding to each target operation task in the executed task queue can be obtained in sequence based on the arrangement order of the at least one target operation task in the executed task queue, and the obtained processing function is executed to obtain the operation result of one target operation task.

[0129] The embodiments of the present application can obtain the operation result of one or more target operation tasks in the executed task queue in sequence without affecting the execution of the other tasks by the working thread, thereby reasonably utilizing the working thread.

[0130] In some embodiments, the target IO operation type corresponding to the IO operation can be determined each time the IO operation is executed in S41; the target callback function matched with the target IO operation type is selected from the at least one callback function based on the mapping relationship between the predefined callback functions and the IO operation types; the target callback function is used as the processing function corresponding to the IO operation, and the processing function is registered to the operating system.

[0131] Among them, for different IO operation types, corresponding callback functions can be defined in advance, such as IO operation types including data read operation, data write operation in file IO operation, and data receiving operation, data sending operation in network IO operation, etc. The target IO operation type can be one of the above types. For example, when performing an IO operation, if the IO operation is a data read operation, the callback function corresponding to the data read operation can be selected as the processing function corresponding to the IO operation, and the processing function is registered with the operating system, so that the operating system can call the processing function after performing the IO operation to pass the corresponding operation result.

[0132] In a specific implementation, when the data processing method of the embodiment of the application is executed by using the Rust-based asynchronous event-driven engine, the type-safe callback function can be realized based on Rust. In Rust, the type of the callback function describes the parameter type received by the callback function and the data type returned by the callback function, so that type errors of the callback function that do not match can be captured at the time of compilation, and it is ensured that the types of all callback functions are determined to be correct at the compilation stage and meet the expected signature and behavior. The expected signature refers to the parameter list and return type that a function, method, interface, trait or other callable entity should have, which describes how to call the callback function, including the parameter type passed to it, the number and order of parameters; the expected behavior refers to the code logic inside the callback function, which determines how the callback function processes the input data. Based on Rust, the type-safe callback function can be realized, and the runtime error of the callback function can be reduced.

[0133] When the type-safe callback function is realized in Rust, the generics and traits of Rust can be used. In Rust, a trait defines a set of method signatures, but does not contain specific implementations, and any type that implements these methods can be considered to have the trait. Generics can be used when defining functions, structures, enumerations or traits without specifying specific data types, but using placeholders (usually called type parameters), which can be replaced with specific data types when actually used.

[0134] The following is a specific example to introduce how to realize the callback function based on Rust and how to apply the callback function.

[0135] For example, in Rust, a callback function handle_event that can handle asynchronous events (i.e., operation task completion) is defined through the EventHandler trait, which receives an event parameter of type EpollEvent, which represents the asynchronous event that the callback function needs to handle. Then, an Eventloop structure can be created to represent the event loop, which can store the mapping relationship between the callback function handle_event and the descriptor (corresponding to the operation task). Specifically, the descriptor is mapped to the callback function handle_event through a HashMap.

[0136] In the specific implementation, an EventLoop instance is created, then a file is opened and set to non-blocking mode; then a FileReaderHandler instance is created, which is an implementation of the above-mentioned EventHandler trait, and the FileReaderHandler instance is added to the EventLoop instance. Finally, the run method is called to start the EventLoop instance (i.e., the event loop), which waits for the occurrence of asynchronous events (i.e., operation task completion) in an infinite loop, i.e., there is a target operation task in the executed task queue, and the callback function handle_event corresponding to the target operation task is called.

[0137] The Rust-based asynchronous event-driven engine implementing the data processing method of the embodiments of the present application has the following advantages:

[0138] Non-blocking asynchronous IO operation: when initiating an IO operation, the software program can continue to execute other tasks while waiting for the completion of the IO operation, thereby improving overall performance.

[0139] Cross-platform support: asynchronous IO operations can be performed on different operating systems (such as Linux, Windows, macOS).

[0140] Event-driven: designed around the event loop, it can efficiently handle tens of thousands of concurrent connections, which is beneficial for software programs to handle a large number of concurrent IO operations.

[0141] High performance: using efficient IO multiplexing technology such as epoll in Linux, kqueue in macOS, and IOCP in Windows, a large number of concurrent IO operations can be efficiently handled without introducing thread context switching overhead, which is beneficial for building high-throughput network services and software programs.

[0142] Memory safety: Rust's ownership and borrowing rules ensure memory safety without the intervention of a garbage collector, allowing for the risk of memory leaks and dangling pointers.

[0143] Type safety: Rust's strong type system reduces the likelihood of runtime errors. Catching errors at compile time can avoid many common concurrency and asynchronous programming issues.

[0144] Zero-cost abstractions: Rust can use high-level abstractions without affecting code performance, making it easier to write asynchronous code while maintaining operating system performance.

[0145] In the embodiments of the present application, each time an IO operation is performed, a target callback function matching the target IO operation type corresponding to the IO operation can be selected as a processing function corresponding to the IO operation, so that the operation result of the IO operation can be accurately obtained based on the processing function subsequently.

[0146] In some embodiments, considering that the number of operation tasks in the to-be-executed task queue can be large, in order to prevent the operation tasks from being left for a long time, in each set period, when it is determined that any operation task in the to-be-executed task queue is in an unfinished state within a set time length during the process of traversing each operation task in the to-be-executed task queue, task wake-up information for the operation task can be sent to the operating system; wherein the task wake-up information is used to instruct to process the operation task.

[0147] Specifically, a delay task of the operation task can be created to record the delay execution time length of the operation task, and when the delay execution time length reaches the set time length, the operation task can be woken up.

[0148] In the embodiments of the present application, in order to ensure that each operation task is executed, in each set period, it is checked whether there is an operation task that has not been executed for a long time in the to-be-executed task queue, and if there is, the operation task can be woken up, so that no operation task is left for a long time, and the scheduling fairness of each operation task in the to-be-executed task queue is ensured.

[0149] In some embodiments, when at least two operation tasks in the to-be-executed operation queue satisfy the set merging condition in S41, the at least two operation tasks are merged to obtain a merged operation task. Wherein the set merging condition includes at least one of the following:

[0150] The IO operation types are the same;

[0151] The targeted IO resources are the same.

[0152] In creating each IO operation corresponding operation task, the IO operation type and the IO resource targeted by the operation task can be determined. The IO operation type includes data read operation in file IO operation, data write operation, data receiving operation in network IO operation, data sending operation, etc. The IO resource includes file, network socket, etc.

[0153] The following describes several possible cases of merging at least two operation tasks.

[0154] In one case, at least two operation tasks with the same IO operation type can be merged.

[0155] For example, for file IO operation, when at least two operation tasks are both data read operation, these operation tasks can be merged into one merged operation task; or when at least two operation tasks are both data write operation, these operation tasks can be merged into one merged operation task.

[0156] For example, for network IO operation, when at least two operation tasks are both data receiving operation, these operation tasks can be merged into one merged operation task; or when at least two operation tasks are both data sending operation, these operation tasks can be merged into one merged operation task.

[0157] In another case, at least two operation tasks targeting the same IO resource can be merged.

[0158] For example, for network IO operation, when at least two operation tasks are both data receiving operation or data sending operation targeting the same network socket, these operation tasks can be merged into one merged operation task.

[0159] For example, for file IO operation, when at least two operation tasks are both data read operation or data write operation targeting the same file, these operation tasks can be merged into one merged operation task.

[0160] In yet another case, at least two operation tasks with the same IO operation type and targeting the same IO resource can be merged.

[0161] For example, for network IO operation, when at least two operation tasks are both data receiving operation targeting the same network socket, these operation tasks can be merged into one operation task; or when at least two operation tasks are both data sending operation targeting the same network socket, these operation tasks can be merged into one merged operation task.

[0162] For example, for file IO operations, when at least two operation tasks are both data read operations on the same file, the operation tasks can be combined into one operation task; or when at least two operation tasks are both data write operations on the same file, the operation tasks can be combined into one combined operation task.

[0163] As shown in FIG. 8, the to-be-executed task queue includes operation task 1 to operation task 8. Operation task 1 and operation task 3 satisfy the combination condition and are combined into a combined operation task. Operation task 4 and operation task 6 satisfy the combination condition and are also combined into a combined operation task. The to-be-executed task queue after combination is: combined operation task 1, operation task 2, combined operation task 2, operation task 7, and operation task 8. Figure 7

[0164] It can be understood that a combined operation task corresponds to at least two operation tasks, each operation task corresponds to a processing function and a descriptor, and when the processing status of the combined operation task is queried, the processing status of the at least two operation tasks can be queried based on the descriptors of the at least two operation tasks. When the at least two operation tasks in the combined operation task are completed, the combined operation task is added to the executed task queue, and the at least two processing functions corresponding to the combined operation task are added to the executed task queue.

[0165] In the embodiments of the present application, at least two operation tasks in the to-be-executed operation queue that satisfy the set combination condition are combined, which can reduce the number of operation tasks in the to-be-executed operation queue, improve the query efficiency of each operation task in the to-be-executed operation queue, and save processing resources consumed in the query process.

[0166] In some embodiments, in order to better schedule and execute each operation task in the to-be-executed task queue, a priority information can be set for each operation task in the to-be-executed task queue according to a set priority rule, and each operation task can be sorted according to the priority information.

[0167] The set priority rule can be set as needed. For example, an operation task with a higher requirement for completion time can be given a higher priority. For example, an operation task related to real-time data processing or user interaction task usually needs to be responded quickly, and thus should be executed preferentially. The priority information can be a priority level, for example, different values represent different priority levels, and the higher the value, the higher the priority level. Optionally, each operation task in the to-be-executed task queue can carry state information, such as an ID of the operation task and priority information.

[0168] As shown in FIG. 8, the to-be-executed task queue includes operation task 1 to operation task 8. Operation task 1 and operation task 3 satisfy the combination condition and are combined into a combined operation task. Operation task 4 and operation task 6 satisfy the combination condition and are also combined into a combined operation task. The to-be-executed task queue after combination is: combined operation task 1, operation task 2, combined operation task 2, operation task 7, and operation task 8. Figure 8 ​As shown, assuming priority information is represented by priority levels, such as 1, 2, 3, the larger the value, the higher the priority level. The queue of tasks to be executed includes operation tasks 1 to 8 (arranged from right to left). Operation tasks 1 to 8 are arranged in descending order of priority, resulting in operation task 3 (priority level 3), operation task 5 (priority level 3), operation task 1 (priority level 2), operation task 4 (priority level 2), operation task 7 (priority level 2), operation task 2 (priority level 1), operation task 6 (priority level 1), and operation task 8 (priority level 1). Among these, operation tasks with the same priority level can be arranged according to the order in which they were added.

[0169] Each time an I / O operation request is sent to the operating system, the priority information of the corresponding I / O operation (i.e., the priority information of the operation task) can be carried in the I / O operation request so that the operating system can execute multiple I / O operations according to their priorities.

[0170] Furthermore, during the execution of each operation task in the queue of tasks to be executed, the processing status of each operation task can be queried in turn according to the priority order of each operation task.

[0171] In this embodiment of the application, priority information is set for each operation task in the queue of tasks to be executed according to the set priority rules, which can better schedule and execute each operation task in the queue of tasks to be executed.

[0172] The following is combined Figure 9 The overall logic of the data processing method in the embodiments of this application will be described by way of example.

[0173] like Figure 9 As shown, when a software program initiates an I / O operation to the operating system, it can create an operation task corresponding to the I / O operation, register the processing function of the I / O operation with the operating system, and then add the operation task corresponding to the I / O operation to the queue of tasks to be executed. The operating system will execute the I / O operations in the queue. During the execution of the I / O operation, after the operating system completes the execution of an I / O operation, it will call the processing function of the I / O operation and pass the operation result to the processing function. The software program can query the processing status of each operation task in the queue of tasks to be executed, add the completed target operation tasks to the queue of executed tasks, and then execute the processing functions corresponding to the target operation tasks in sequence to obtain the operation results of the corresponding I / O operations.

[0174] Therefore, the embodiments of this application enable software programs to perform concurrent I / O operations asynchronously. Each time an I / O operation is performed, there is no need to wait for the I / O operation to complete before other tasks can continue to be executed. Furthermore, when a large number of I / O operations need to be processed, the processing efficiency of I / O operations is improved.

[0175] The following exemplary introduces the specific application scenario of the data processing method of the embodiment of the application.

[0176] In practical applications, servers often need to handle a large number of concurrent network requests, involving scenarios that handle a large number of asynchronous I / O operations. For example, the following scenarios in which the server of an e-commerce website performs a large number of asynchronous I / O operations:

[0177] I. User login. When a user sends a login request to an e-commerce website, the server needs to read user information from the database for verification. Since the database query operation may be time-consuming, the server usually queries the database asynchronously to read user information, that is, performs file IO operations asynchronously; at this time, the server can use the data processing method of the embodiment of the application when performing file IO operations for each login request, so that it can continue to process other requests during the waiting period for the return of user information.

[0178] II. Product search. When a user searches for products on an e-commerce platform, the server needs to retrieve matching products from the product catalog. This usually involves complex queries on the product database, that is, file IO operations, which the server performs asynchronously to improve response speed.

[0179] III. Cart operation. When a user adds a product to the shopping cart or modifies the quantity of products in the shopping cart, the server needs to update the cart information in the database asynchronously, that is, perform file IO operations asynchronously, while returning the latest cart status to the user.

[0180] IV. Payment processing. Payment operations involve communication with third-party payment gateways, which often takes a long time. The server initiates payment requests asynchronously, and then notifies the server of the payment status through callback or polling when the payment request is completed, that is, performs network IO operations asynchronously.

[0181] V. Log recording and data analysis. The server needs to record user operations and system status, and these log recording operations can be implemented by performing file I / O operations asynchronously to avoid affecting real-time response.

[0182] The server of an e-commerce platform can use the data processing method of the embodiment of the application to handle asynchronous IO operations in the above scenarios, so as to simultaneously process multiple network requests, improve the response speed and throughput of the system, and thus provide better user experience and system performance.

[0183] Based on the same inventive concept, the embodiment of the application also provides a data processing device, the principle of which solves the problem similar to the above-mentioned method of the embodiment, and therefore the implementation of the device can be referred to the implementation of the above-mentioned method, and the repeated parts will not be described again.

[0184] As Figure 10 shown, it is a structural schematic diagram of the data processing apparatus 1000, which can include:

[0185] The creating unit 1001 is configured to create a processing function corresponding to an input / output (IO) operation each time the IO operation is performed, and add an operation task corresponding to the IO operation to a to-be-executed task queue; wherein the processing function is used to obtain an operation result corresponding to the operation task after the operation task is executed.

[0186] The querying unit 1002 is configured to query a processing state of each operation task in the to-be-executed task queue in a process of executing each operation task in the to-be-executed task queue.

[0187] The transferring unit 1003 is configured to add a target operation task in the to-be-executed task queue to an executed task queue when it is queried that the processing state of the target operation task is completed.

[0188] The processing unit 1004 is configured to perform the following operations for at least one target operation task in the executed task queue: executing a processing function corresponding to one target operation task to obtain an operation result of the one target operation task.

[0189] In the embodiment of the present application, a processing function of an IO operation is created each time the IO operation is performed, which is used to subsequently obtain a processing result of an operation task corresponding to the IO operation. In this way, other tasks, such as initiating a new IO operation or other operations, can be performed without waiting for the IO operation to be completed. In addition, an operation task corresponding to each IO operation is added to a to-be-executed task queue, thereby facilitating scheduling of multiple operation tasks in the to-be-executed task queue, transferring a completed target operation task in the multiple operation tasks to an executed task queue, and thereby facilitating calling of a processing function of the completed target operation task in sequence to obtain a processing result of the target operation task. Therefore, the embodiment of the present application can avoid blocking of a work thread of a software program when an IO operation is initiated, save processing resources, and improve processing efficiency of the IO operation when a large number of IO operations need to be processed.

[0190] In an optional implementation, when a processing function corresponding to an input / output (IO) operation is created each time the IO operation is performed, the creating unit 1001 is specifically configured to:

[0191] Determine a target IO operation type corresponding to the IO operation each time the IO operation is performed;

[0192] Select a target callback function matched with the target IO operation type from at least one callback function based on a mapping relationship between the callback functions and the IO operation types;

[0193] The target callback function is registered as a processing function corresponding to the IO operation to the operating system.

[0194] In an optional implementation, when adding the operation task corresponding to the IO operation to the to-be-executed task queue, the creating unit 1001 is specifically configured to:

[0195] create an operation task corresponding to the IO operation, and initiate an IO operation request corresponding to the IO operation to the operating system;

[0196] obtain a descriptor of an IO resource required for executing the IO operation from the operating system, and register a processing function corresponding to the IO operation to the operating system; wherein the descriptor is used to identify the IO resource;

[0197] add the operation task carrying the descriptor to the to-be-executed task queue.

[0198] In an optional implementation, the apparatus further includes:

[0199] The registering unit is configured to register a descriptor corresponding to the operation task to an IO multiplexing interface of the operating system; wherein the IO multiplexing interface is used to detect a state of at least one descriptor, so that the operating system executes a corresponding operation task according to the state of each descriptor to obtain a processing state of the operation task.

[0200] The querying unit 1002 is specifically configured to:

[0201] In the process of executing each operation task in the to-be-executed task queue by the operating system, every time a set period is reached, each operation task in the to-be-executed task queue is sequentially traversed; wherein each time an operation task is traversed, a processing state of the corresponding operation task is queried from the IO multiplexing interface based on a descriptor corresponding to one operation task.

[0202] In an optional implementation, the apparatus further includes:

[0203] The waking unit is configured to send task wake-up information for any operation task to the operating system when it is determined that any operation task in the to-be-executed task queue is in an unfinished state within a set time length.

[0204] The task wake-up information is used to instruct to process the operation task.

[0205] In an optional implementation, the transferring unit 1003 is specifically configured to:

[0206] When the processing state of the target operation task in the to-be-executed task queue is queried to be completed, the target operation task is deleted from the to-be-executed task queue;

[0207] Add the target operation task and the processing function corresponding to the target operation task to the executed task queue.

[0208] In an optional implementation, the target operation tasks in the executed task queue are arranged in the order of completion.

[0209] The processing unit 1004 is specifically configured to:

[0210] When it is determined that the worker thread for processing the target operation task is idle, the processing function corresponding to each target operation task in the executed task queue is sequentially obtained based on the arrangement order of the at least one target operation task in the executed task queue, and the obtained processing function is executed to obtain an operation result of a target operation task.

[0211] In an optional implementation, the apparatus further includes:

[0212] The merging unit is configured to merge at least two operation tasks in the to-be-executed operation queue when the at least two operation tasks satisfy a set merging condition, to obtain a merged operation task.

[0213] The set merging condition includes at least one of the following:

[0214] The IO operation types are the same;

[0215] The IO resources targeted are the same.

[0216] In an optional implementation, the apparatus further includes:

[0217] The setting unit is configured to set priority information for each operation task in the to-be-executed task queue according to a set priority rule, and sort the operation tasks according to the priority information.

[0218] The query unit 1002 is specifically configured to:

[0219] In the process of executing each operation task in the to-be-executed task queue, the processing state of each operation task is sequentially queried according to the priority arrangement order of the operation tasks.

[0220] For the convenience of description, each part is described as a module (or unit) according to the function. Of course, the functions of the modules (or units) can be implemented in the same or multiple software or hardware in the implementation of the present application.

[0221] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0222] After introducing the data processing method and device of the example embodiments of the present application, next, the electronic device according to another example embodiment of the present application is introduced.

[0223] Based on the same inventive concept as the above method embodiments, the present embodiments also provide an electronic device. In an embodiment, the electronic device can be a server, such as the server 110 shown in Figure 1 In this embodiment, the structure of the electronic device can be as shown in Figure 11 includes a memory 1101, a communication module 1103, and one or more processors 1102.

[0224] The memory 1101 is used to store computer programs executed by the processor 1102. The memory 1101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and programs required for running instant messaging functions, etc.; the data storage area can store various instant messaging information and operation instruction sets, etc.

[0225] The memory 1101 can be a volatile memory (volatile memory), such as a random access memory (random-access memory, RAM); the memory 1101 can also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory, a hard disk drive (hard disk drive, HDD) or a solid-state drive (solid-state drive, SSD); or the memory 1101 is any other medium that can be used to carry or store desired computer programs in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory 1101 can be a combination of the above memories.

[0226] The processor 1102 can include one or more central processing units (central processing unit, CPU) or digital processing units, etc. The processor 1102 is used to implement the above data processing method when invoking the computer program stored in the memory 1101.

[0227] The communication module 1103 is configured to communicate with the terminal device and other servers.

[0228] The specific connection medium between the memory 1101, the communication module 1103 and the processor 1102 is not limited in the embodiments of the present application. In the embodiments of the present application, the memory 1101 and the processor 1102 are connected through the bus 1104, and the bus 1104 is described by a thick line in the embodiments of the present application. The connection mode between other components is only schematically described, and is not limited. Figure 11 Figure 11 The bus 1104 can be divided into an address bus, a data bus, a control bus and the like. For the convenience of description, only one thick line is used to describe the bus 1104 in the embodiments of the present application, but it is not described that there is only one bus or only one type of bus. Figure 11

[0229] The electronic device 1200 according to the embodiments of the present application will be described below with reference to FIG. 12. Figure 12 The electronic device 1200 is only one example and should not be taken as limiting the functionality or use of embodiments of the present application. Figure 12 As shown in FIG. 12, the electronic device 1200 is in the form of a general electronic device. The components of the electronic device 1200 can include, but are not limited to, at least one processing unit 1212, at least one storage unit 1220, and a bus 1230 connecting different system components, including the storage unit 1220 and the processing unit 1212.

[0230] Figure 12 The bus 1230 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor or local bus using any of a variety of bus structures.

[0231] The storage unit 1220 can include a readable medium in the form of volatile memory, such as a random access memory (RAM) 1221 and / or a cache memory 1222, and can further include a read-only memory (ROM) 1223.

[0232] The storage unit 1220 can further include a program / utility 1225 having a set of program modules 1224, including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which can include implementation of a network environment, alone or in combination with some of the examples.

[0233] The storage unit 1220 can further include a program / utility 1225 having a set of program modules 1224, including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which can include implementation of a network environment, alone or in combination with some of the examples.

[0234] ​​​The electronic device 1200 can also communicate with one or more external devices 1240 such as a keyboard or a pointing device, by way of Input / Output (I / O) interface 1250. Further, the electronic device 1200 can communicate with one or more devices that enable user interaction with the electronic device 1200, and / or one or more devices that enable communication of the electronic device 1200 with one or more other electronic devices. For example, the electronic device 1200 can communicate with one or more devices via an appropriate I / O interface 1250. Such communication can be facilitated by way of an I / O interface 1250. Additionally, the electronic device 1200 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the public network, such as the Internet, by way of the network adapter 1260. As illustrated, the network adapter 1260 communicates with other modules of the electronic device 1200 via the bus 1230. It should be appreciated that other hardware and / or software modules can be used in conjunction with the electronic device 1200, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0235] In some possible implementation, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium includes a computer program, when the computer program is executed on an electronic device, the computer program is used to make the electronic device execute the steps of the data processing method of the embodiment of the present application, such as Figure 4 shown in the figure.

[0236] In some possible implementation, an embodiment of the present application provides a computer program product, the computer program product includes a computer program, the computer program is stored in a computer readable storage medium; when the processor of the electronic device reads the computer program from the computer readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of any one of the above data processing methods, for example, the electronic device can execute the steps as Figure 4 shown in the figure.

[0237] The computer program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0238] The computer program product of the embodiments of the present application can be embodied in a compact disc read only memory (CD-ROM), and include a computer program. However, the program product of the present application is not limited thereto, and in the present document, the readable storage media can be any tangible media that contain or store a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0239] The readable signal media can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the readable computer program is embodied. Such a propagated data signal can take a variety of forms, including but not limited to electro-magnetic signal, optical signal, or any suitable combination thereof. The readable signal media can also be any readable media that can be used to carry, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0240] The computer program embodied on the readable media can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber cable, RF, and the like, or any suitable combination thereof.

[0241] The computer program for performing the operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer program can execute entirely on the user's electronic device, partly on the user's electronic device, as a stand-alone software package, partly on the user's electronic device and partly on a remote electronic device or entirely on the remote electronic device or server. In the latter scenario, the remote electronic device can be connected to the user's electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external electronic device (for example, through the Internet using an Internet Service Provider). In some embodiments of the present application, electronic device refers to Portable Computer, Personal Digital Assistant (PDA), Ultra Mobile Personal Computer (UMPC), Mobile Phone, Smart Phone, or any other portable electronic device.

[0242] It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units for embodiment.

[0243] Moreover, although the operations of the method(s) herein can be described in a particular, sequential order, this order is not meant to be a limitation and

[0244] Those of skill in the art would understand that embodiments of the present application can be provided as a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0245] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing unit or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0246] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0247] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0248] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that such additions and modifications be included within the scope of the application. It is the following claims, including any amendments thereto, which define the scope of the application.

[0249] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A data processing method, characterized in that, The method includes: Each time an input / output (I / O) operation is performed, a processing function corresponding to the I / O operation is created, and the operation task corresponding to the I / O operation is added to the task queue to be executed; wherein, the processing function is used to obtain the corresponding operation result after the operation task is completed; During the execution of each operation task in the queue of tasks to be executed, the processing status of each operation task in the queue of tasks to be executed is queried. When the processing status of the target operation task in the queue of tasks to be executed is found to be completed, the target operation task is added to the queue of executed tasks. For at least one target operation task in the executed task queue, perform the following operations respectively: execute the processing function corresponding to the target operation task to obtain the operation result of the target operation task.

2. The method according to claim 1, characterized in that, Each time an input / output I / O operation is performed, a corresponding processing function is created, including: Each time the IO operation is executed, the target IO operation type corresponding to the IO operation is determined; Based on the predefined mapping relationship between callback functions and I / O operation types, a target callback function that matches the target I / O operation type is selected from at least one callback function; The target callback function is used as the processing function corresponding to the IO operation.

3. The method according to claim 1, characterized in that, Adding the operation task corresponding to the IO operation to the queue of tasks to be executed includes: Create the operation task corresponding to the IO operation and send the IO operation request corresponding to the IO operation to the operating system; Obtain descriptors of the IO resources required to perform the IO operation from the operating system, and register the processing function corresponding to the IO operation with the operating system; wherein, the descriptor is used to identify the IO resource; The operation task carrying the descriptor is added to the queue of tasks to be executed.

4. The method according to claim 3, characterized in that, After adding the operation task carrying the descriptor to the queue of tasks to be executed, the method further includes: Register the descriptor corresponding to the operation task with the I / O multiplexing interface of the operating system; wherein, the I / O multiplexing interface is used to detect the state of at least one descriptor so that the operating system executes the corresponding operation task according to the state of each descriptor and obtains the processing state of the operation task. The step of querying the processing status of each operation task in the queue of tasks to be executed during the execution of each operation task includes: During the execution of each operation task in the queue of tasks to be executed by the operating system, each operation task in the queue of tasks to be executed is traversed sequentially at set intervals; wherein, for each operation task, the processing status of the corresponding operation task is queried from the IO multiplexing interface based on the descriptor corresponding to the operation task.

5. The method according to claim 4, characterized in that, After sequentially traversing each operation task in the queue to be executed at predetermined intervals, the process further includes: When it is determined that any operation task in the queue of tasks to be executed is in an incomplete state within a set time period, a task wake-up message for any operation task is sent to the operating system. The task wake-up information is used to instruct the processing of the operation task.

6. The method according to any one of claims 1 to 5, characterized in that, When the processing status of the target operation task in the pending task queue is found to be completed, the target operation task is added to the executed task queue, including: When the processing status of the target operation task in the queue of tasks to be executed is found to be completed, the target operation task is deleted from the queue of tasks to be executed. The target operation task and its corresponding processing function are added to the executed task queue.

7. The method according to claim 6, characterized in that, The multiple target operation tasks in the executed task queue are arranged in the order of completion; Then, for at least one target operation task in the executed task queue, the following operations are performed respectively: executing the processing function corresponding to a target operation task to obtain the operation result of the target operation task, including: When it is determined that the worker thread used to process the target operation task is idle, based on the order of at least one target operation task in the executed task queue, the processing function corresponding to each target operation task is obtained from the executed task queue in sequence, and the obtained processing function is executed to obtain the operation result of a target operation task.

8. The method according to any one of claims 1 to 5, characterized in that, The method further includes: When at least two operation tasks in the queue to be executed meet the set merging conditions, the at least two operation tasks are merged to obtain merged operation tasks. The set merging conditions include at least one of the following: The IO operation types are the same; The same IO resources are being targeted.

9. The method according to any one of claims 1 to 5, characterized in that, The method further includes: According to the set priority rules, priority information is set for each operation task in the queue of tasks to be executed, and the operation tasks are sorted according to the priority information; The step of querying the processing status of each operation task in the queue of tasks to be executed during the execution of each operation task includes: During the execution of each operation task in the queue of tasks to be executed, the processing status of each operation task is queried in turn according to the priority order of each operation task.

10. A data processing apparatus, characterized in that, The device includes: A creation unit is used to create a processing function corresponding to each input / output (I / O) operation and add the operation task corresponding to the I / O operation to a queue of tasks to be executed; wherein, the processing function is used to obtain the corresponding operation result after the operation task is completed; The query unit is used to query the processing status of each operation task in the queue of tasks to be executed during the execution of each operation task in the queue of tasks to be executed. The transfer unit is used to add the target operation task to the executed task queue when the processing status of the target operation task in the queue of tasks to be executed is found to be completed. The processing unit is configured to perform the following operations for at least one target operation task in the executed task queue: execute the processing function corresponding to a target operation task, and obtain the operation result of the target operation task.

11. The apparatus according to claim 10, characterized in that, When creating the processing function corresponding to each input / output I / O operation, the creation unit is specifically used for: Each time the IO operation is executed, the target IO operation type corresponding to the IO operation is determined; Based on the predefined mapping relationship between callback functions and I / O operation types, a target callback function that matches the target I / O operation type is selected from at least one callback function; The target callback function is used as the processing function corresponding to the IO operation, and the processing function is registered with the operating system.

12. The apparatus according to claim 10, characterized in that, When adding the operation task corresponding to the IO operation to the queue of tasks to be executed, the creation unit is specifically used for: Create an operation task corresponding to the IO operation and send an IO operation request corresponding to the operation task to the operating system; The system obtains the descriptor of the IO resource required to perform the IO operation from the operating system, and adds the operation task carrying the descriptor to the queue of tasks to be executed; wherein the descriptor is used to identify the IO resource.

13. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any one of the methods described in claims 1 to 9.

14. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1 to 9.

15. A computer program product, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any one of claims 1 to 9.

Images