Data processing method and device, electronic equipment and storage medium

Abstract

The present application relates to data processing, and provides a data processing method, device, electronic equipment and storage medium. The method divides a preset process pool into a first process pool for input / output operations and a second process pool for computing operations, and respectively creates corresponding first and second work queues. When a data processing request is received, it is determined whether the data to be processed is complete. If so, the data to be processed is encapsulated as a computing task and stored in the second work queue. It is determined whether there is an idle process in the second process pool. If so, the idle process is used to read the computing task from the second work queue and perform computing processing to obtain a computing result. Then, the computing result is stored in the first work queue. It is determined whether there is an idle process in the first process pool. If so, the idle process is used to read the computing result from the first work queue. The present application can provide sufficient parallel processing capability and reduce the consumption of system resources.

Description

TECHNICAL FIELD

[0001] The present application relates to data processing, in particular to a data processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] The current computing service middleware system, the input and output of data and the calculation are in the same process or thread, the input and output operation, the calculation operation and the acquisition of the to-be-calculated data are processed in the same process or thread.

[0003] Since the resources used by the input and output and the calculation are shared, when the performance of the input and output end is asymmetric, the input and output processing will consume a large amount of computing resources, reducing the processing capacity of the system, and since the input and output and the calculation are shared, when a large amount of input and output needs to be processed in a unit of time, the resource occupation ratio used by the input and output will be much larger than that used by the calculation, and in a large concurrency, the system will allocate more resources to coordinate the switching of processes or threads. SUMMARY

[0004] In view of the above, the present application provides a data processing method, device, electronic equipment and storage medium, which aims to solve the technical problem that the input and output operation and the calculation operation are processed in the same process, resulting in large consumption of system resources.

[0005] To achieve the above purpose, the present application provides a data processing method, which comprises:

[0006] The creating step: dividing a preset process pool into a first process pool for input and output operation and a second process pool for calculation operation, respectively creating a first work queue corresponding to the first process pool and a second work queue corresponding to the second process pool;

[0007] The receiving step: receiving a request for data processing issued by a client, acquiring the to-be-processed data in the request, judging whether the to-be-processed data is complete based on a pre-configured data protocol, when the to-be-processed data is complete, encapsulating the to-be-processed data as a calculation task, and storing the calculation task to the second work queue;

[0008] The processing step: judging whether there is an idle process in the second process pool, when it is judged that there is an idle process in the second process pool, reading the calculation task from the second work queue by using the idle process, performing calculation processing on the calculation task based on a preset processing rule to obtain a calculation result, and storing the calculation result to the first work queue; and

[0009] a feedback step of determining whether there is an idle process in the first process pool, and when it is determined that there is an idle process in the first process pool, reading the calculation result from the first work queue by using the idle process and feeding back the calculation result to the client.

[0010] Preferably, the receiving step further comprises:

[0011] When it is determined that the to-be-processed data is incomplete, it is determined whether there is an idle process in the first process pool, when it is determined that there is no idle process in the first process pool, the to-be-processed data is stored to the first work queue, and when it is determined that there is an idle process in the first process pool, the incomplete data corresponding to the to-be-processed data is received by using the idle process.

[0012] Preferably, after the feedback step, the method further comprises:

[0013] It is determined whether the first work queue stores to-be-processed data or to-be-fed-back calculation result, and when it is determined that the first work queue does not store to-be-processed data and to-be-fed-back calculation result, the idle process in the first process pool is set to a preset state.

[0014] Preferably, after the processing step, the method further comprises:

[0015] It is determined whether the second work queue stores to-be-processed calculation task, and when it is determined that the second work queue does not store to-be-processed calculation task, the idle process in the second process pool is set to a preset state.

[0016] When it is determined that the second work queue stores to-be-processed calculation task, the to-be-processed calculation task is read from the second work queue based on the idle process in the second process pool, and calculation processing is performed on the calculation task.

[0017] Preferably, the feedback step further comprises:

[0018] When it is determined that there is no idle process in the first process pool, the calculation result is encapsulated into a transmission task, the transmission task is stored to the first work queue, and a transmission operation is performed on the transmission task when there is an idle process in the first process pool.

[0019] Preferably, the calculation processing on the calculation task based on the preset processing rule to obtain the calculation result comprises:

[0020] An identifier of the calculation task is obtained, a calculation processing rule corresponding to the calculation task is searched from a pre-configured mapping relationship table based on the identifier, and calculation processing is performed on the calculation task based on the calculation processing rule to obtain the calculation result.

[0021] To achieve the above object, the application further provides a data processing device, characterized in that the device comprises:

[0022] a creating module: dividing a preset process pool into a first process pool for input / output operations and a second process pool for calculation operations, and respectively creating a first work queue corresponding to the first process pool and a second work queue corresponding to the second process pool;

[0023] a receiving module: receiving a request for data processing sent by a client, obtaining to-be-processed data in the request, judging whether the to-be-processed data is complete based on a preconfigured data protocol, when the to-be-processed data is complete, encapsulating the to-be-processed data into a calculation task, and storing the calculation task to the second work queue;

[0024] a processing module: judging whether there is an idle process in the second process pool, when it is judged that there is an idle process in the second process pool, reading the calculation task from the second work queue by using the idle process, performing calculation processing on the calculation task based on a preset processing rule to obtain a calculation result, and storing the calculation result to the first work queue; and

[0025] a feedback module: judging whether there is an idle process in the first process pool, when it is judged that there is an idle process in the first process pool, reading the calculation result from the first work queue by using the idle process, and feeding back the calculation result to the client.

[0026] Preferably, the receiving module further comprises:

[0027] when it is judged that the to-be-processed data is not complete, judging whether there is an idle process in the first process pool, when it is judged that there is no idle process in the first process pool, storing the to-be-processed data to the first work queue, and when it is judged that there is an idle process in the first process pool, receiving incomplete data corresponding to the to-be-processed data by using the idle process.

[0028] To achieve the above object, the application further provides an electronic device, characterized in that the electronic device comprises:

[0029] at least one processor; and

[0030] a memory in communication connection with the at least one processor; wherein

[0031] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any step in the data processing method as described above.

[0032] To achieve the above object, the application further provides a computer readable storage medium, wherein the computer readable storage medium comprises a data processing program, and the data processing program is executed by a processor to realize any step of the data processing method.

[0033] The data processing method, device, electronic equipment and storage medium provided by the application store input / output operations and calculation operations into the pre-divided process pool respectively, perform asynchronous processing, provide sufficient parallel processing capability, fully utilize the multi-core parallel processing capability of modern processors, eliminate the lock introduced due to concurrency when the process pool acquires processing tasks and storage tasks, and reduce the consumption of system resources caused by the contention of the lock. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A schematic diagram of a preferred embodiment of an electronic equipment for realizing the data processing method of the application;

[0035] Figure 2 A module schematic diagram of a preferred embodiment of the data processing device of the application;

[0036] Figure 3 A flow chart of a preferred embodiment of the data processing method of the application;

[0037] The realization of the object, functional features and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0038] In order to make the object, technical scheme and advantages of the application more clear, the application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and not used to limit the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0039] Referring to Figure 1 The schematic diagram of a preferred embodiment of an electronic equipment 1 of the application is shown.

[0040] The electronic device 1 includes, but is not limited to, a memory 11, a processor 12, a display 13, and a network interface 14. The electronic device 1 is connected to a network through the network interface 14 to obtain raw data. The network can be an intranet, the Internet, a Global System of Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, Wi-Fi, a telephone network, or other wireless or wired networks.

[0041] The memory 11 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, or the like. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a hard disk or a memory of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like. Of course, the memory 11 can include both an internal storage unit and an external storage device of the electronic device 1. In this embodiment, the memory 11 is generally used to store an operating system and various application software installed in the electronic device 1, such as program codes of the data processing program 10, and the like. In addition, the memory 11 can also be used to temporarily store various data that has been output or will be output.

[0042] The processor 12 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication, and the like. In this embodiment, the processor 12 is used to run program codes or process data stored in the memory 11, such as running program codes of the data processing program 10, and the like.

[0043] The display 13 can be referred to as a display screen or a display unit. In some embodiments, the display 13 can be an LED display, a liquid crystal display, a touch liquid crystal display, an Organic Light-Emitting Diode (OLED) touch screen, or the like. The display 13 is used to display information processed in the electronic device 1 and to display a visualized user interface, for example, to display results of data statistics.

[0044] The network interface 14 can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0045] Figure 1 Only the electronic device 1 with components 11-14 and the data processing program 10 is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead.

[0046] Optionally, the electronic device 1 can also include a user interface, which can include a display, an input unit such as a keyboard, and optionally a standard wired interface, a wireless interface. In some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, an Organic Light-Emitting Diode (OLED) touch screen, or the like. The display can also be appropriately referred to as a display screen or a display unit, and is used to display information processed in the electronic device 1 and to display a visualized user interface.

[0047] The electronic device 1 can also include a radio frequency (RF) circuit, a sensor, an audio circuit, and the like, which are not described here.

[0048] In the above embodiments, when the processor 12 executes the data processing program 10 stored in the memory 11, the following steps can be implemented:

[0049] The creating step: dividing a preset process pool into a first process pool for input / output operations and a second process pool for calculation operations, respectively creating a first work queue corresponding to the first process pool and a second work queue corresponding to the second process pool;

[0050] The receiving step: receiving a request for data processing issued by a client, obtaining the to-be-processed data in the request, judging whether the to-be-processed data is complete based on a pre-configured data protocol, when the to-be-processed data is complete, encapsulating the to-be-processed data as a calculation task, and storing the calculation task to the second work queue;

[0051] Processing steps: Determine whether there is an idle process in the second process pool. If an idle process exists, use that idle process to read the computation task from the second work queue, perform computation processing on the computation task based on preset processing rules to obtain the computation result, and store the computation result in the first work queue; and

[0052] Feedback step: Determine whether there is an idle process in the first process pool. When it is determined that there is an idle process in the first process pool, use the idle process to read the calculation result from the first work queue and feed the calculation result back to the client.

[0053] The storage device can be the memory 11 of the electronic device 1, or it can be other storage devices that are communicatively connected to the electronic device 1.

[0054] For a detailed explanation of the above steps, please refer to the following: Figure 2 Functional block diagram of the embodiment of the data processing device 100 and Figure 3 Description of the flowchart of an embodiment of the data processing method.

[0055] Reference Figure 2 The diagram shown is a functional block diagram of the data processing device 100 of the present invention.

[0056] The data processing device 100 of this invention can be installed in an electronic device. Depending on the functions implemented, the data processing device may include a creation module 110, a receiving module 120, a processing module 130, and a feedback module 140. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0057] In this embodiment, the functions of each module / unit are as follows:

[0058] The creation module 110 is used to divide the preset process pool into a first process pool for input and output operations and a second process pool for computation operations, and to create a first work queue corresponding to the first process pool and a second work queue corresponding to the second process pool, respectively.

[0059] In the embodiment, the process pool pre-configured by the electronic device is divided into a process pool for input / output operations and a process pool for computing operations, a first work queue corresponding to the input / output operation process pool is created, and a second work queue corresponding to the computing operation process pool is created. After separating and splitting the input / output (IO) operations and the computing operations, the operations are processed respectively, which can improve the parallel processing capability, fully utilize the multi-core parallel processing capability of the modern processor, and divide the process pool into multiple processes. The process is a running activity of a program on a data set in a computer, is a basic unit of resource allocation and scheduling of a system, and is a basis of an operating system structure. In a computer structure designed for threads, a process is a container of a thread.

[0060] The receiving module 120 is configured to receive a data processing request sent by a client, acquire to-be-processed data in the request, determine whether the to-be-processed data is complete based on a pre-configured data protocol, encapsulate the to-be-processed data into a computing task when the to-be-processed data is complete, and store the computing task in the second work queue.

[0061] In the embodiment, a data processing request sent by a client is received, to-be-processed data in the request is acquired, it is determined whether the to-be-processed data is complete based on a pre-configured data protocol, the to-be-processed data is encapsulated into a computing task when the to-be-processed data is complete, and the computing task is stored in the second work queue. When the to-be-processed data is incomplete, the incomplete data is stored in the first work queue, and the complete to-be-processed data is encapsulated into a computing task after the data is received completely. The completeness of the to-be-processed data is determined according to the pre-configured data protocol. If there is no idle process in the computing operation process pool, the computing task can be stored in the second work queue corresponding to the computing operation process pool, and the consumption of a lock can be avoided.

[0062] In one embodiment, the receiving module is further configured to:

[0063] When it is determined that the to-be-processed data is incomplete, it is determined whether there is an idle process in the first process pool, the to-be-processed data is stored in the first work queue when it is determined that there is no idle process in the first process pool, and the incomplete data corresponding to the to-be-processed data is received by using the idle process when it is determined that there is an idle process in the first process pool.

[0064] When the to-be-processed data is incomplete and there is no idle process in the input / output operation process pool, the to-be-processed data is stored in the first work queue, an idle process in the input / output operation process pool actively acquires the to-be-processed data without a lock, the lock contention for process resources in the input / output operation process pool is reduced to a lock-free state, and the problem of low resource utilization caused by lock contention is avoided.

[0065] In an embodiment, the data processing apparatus 100 is further configured to:

[0066] determine whether the first work queue stores data to be processed or a calculation result to be fed back, and set an idle process in the first process pool to a preset state when it is determined that neither the first work queue stores data to be processed nor a calculation result to be fed back. The preset state includes a sleep state.

[0067] The processing module 130 is configured to determine whether there is an idle process in the second process pool, read the calculation task from the second work queue by using the idle process when it is determined that there is an idle process in the second process pool, perform calculation processing on the calculation task based on a preset processing rule to obtain a calculation result, and store the calculation result in the first work queue.

[0068] In the embodiment, it is determined whether there is an idle process in the process pool of the calculation operation, the calculation task is read from the second work queue by using the idle process in the process pool of the calculation operation when there is an idle process in the process pool of the calculation operation, calculation processing is performed on the calculation task based on a preset processing rule to obtain a calculation result, and the obtained calculation result is stored in the first work queue.

[0069] Due to the characteristics of the calculation-intensive calculation operation, the process pool of the divided calculation operation can continuously process without interruption of the calculation pipeline under the condition that the data to be calculated is complete, which avoids the situation that the idle process cannot provide services due to the suspension or interruption of the calculation pipeline, and also avoids the problem of system resource consumption caused by the frequent system resource scheduling due to the suspension or interruption of the process, thereby improving the utilization rate of system resources, wherein the system resources include CPU resources, memory resources, network resources, and the like.

[0070] The second work queue corresponding to the process of the calculation operation is created, the calculation task ready for calculation is stored in the second work queue, the process of the calculation processing actively and lock-free acquires the calculation task, so that the lock contention for process resources in the process pool of the input and output operation is reduced to a lock-free state, and the non-functional consumption of system resources caused by the lock contention is reduced. It should be noted that in the prior art, in order to prevent repeated processing, a certain locking process is required when the task is acquired, so as to prevent inconsistency or repetition of resources.

[0071] Further, the calculation processing on the calculation task based on the preset processing rule to obtain a calculation result includes:

[0072] Obtaining an identifier of the computing task, finding a computing processing rule corresponding to the computing task from a pre-configured mapping relation table based on the identifier, and performing computing processing on the computing task based on the computing processing rule to obtain a computing result.

[0073] In one embodiment, the data processing apparatus 100 is further configured to:

[0074] Determining whether the second work queue stores a computing task to be processed, and setting an idle process in the second process pool to a preset state when it is determined that the second work queue does not store the computing task to be processed.

[0075] When it is determined that the second work queue stores the computing task to be processed, reading the computing task to be processed from the second work queue based on the idle process in the second process pool, and performing computing processing on the computing task. The preset state includes a sleep state.

[0076] The feedback module 140 is configured to determine whether there is an idle process in the first process pool, read the computing result from the first work queue using the idle process in the first process pool when it is determined that there is an idle process in the first process pool, and feed back the computing result to the client.

[0077] In this embodiment, it is determined whether there is an idle process in the process pool of the input / output operation. If there is, the computing result is read from the first work queue using the idle process, and the computing result is fed back to the client. This can eliminate the lock introduced due to concurrency when the process pool acquires a processing task, and reduce resource consumption caused by lock contention.

[0078] Due to the waiting characteristics of the input / output operation, the divided input / output operation process pool can support asynchronous callback processing operations based on the asynchronous input / output mechanism supported by the underlying operating system, solve the consumption of processes or threads caused by input / output waiting, reduce the consumption of system resources caused by the frequent creation, destruction or scheduling of processes or threads under large concurrency, and improve the user's usage rate of system resources. It should be noted that the non-asynchronous IO will wait indefinitely or make the process sleep when the data transfer is not completed, and will continue the subsequent operation only when the data reception is completed, which makes the process resources used by the current IO occupied. The asynchronous IO will use process or thread resources only after the data is received.

[0079] On the divided input / output operation process pool, a corresponding first work queue is set, the input / output ready calculation result is stored in the corresponding first work queue, and the input / output processing process actively and lock-free takes, so that the lock contention of the process or thread resources in the input / output processing process pool is reduced to a lock-free state, and the low resource utilization caused by lock contention is greatly reduced.

[0080] In one embodiment, the feedback module is further configured to:

[0081] When it is determined that there is no idle process in the first process pool, the calculation result is encapsulated into a transmission task, and the transmission task is stored in the first work queue until the transmission task is executed when there is an idle process in the first process pool.

[0082] In addition, the present application also provides a data processing method. Referring to Figure 3 The processor 12 of the electronic device 1 implements the following steps of the data processing method when executing the data processing program 10 stored in the memory 11.

[0083] Step S10: Dividing a preset process pool into a first process pool for input / output operation and a second process pool for calculation operation, and creating a first work queue corresponding to the first process pool and a second work queue corresponding to the second process pool.

[0084] In this embodiment, the process pool pre-configured by the electronic device is divided into a process pool for input / output operation and a process pool for calculation operation, a first work queue corresponding to the process pool for input / output operation is created, and a second work queue corresponding to the process pool for calculation operation is created. After separating and splitting the input / output (IO) operation and the calculation operation, they are processed respectively, which can improve the parallel processing capability, fully utilize the multi-core parallel processing capability of modern processors, and the divided process pool includes multiple processes. A process is a running activity of a program on a data set in a computer, is a basic unit of resource allocation and scheduling of a system, and is the basis of operating system structure. In a computer structure designed for threads, a process is a container of threads.

[0085] Step S20: Receiving a request for data processing issued by a client, obtaining the to-be-processed data in the request, judging whether the to-be-processed data is complete based on a pre-configured data protocol, encapsulating the to-be-processed data as a calculation task when the to-be-processed data is complete, and storing the calculation task in the second work queue.

[0086] In the embodiment, a data processing request sent by a client is received, data to be processed in the request is acquired, it is judged whether the data to be processed is complete based on a pre-configured data protocol, when the data to be processed is complete, the data to be processed is encapsulated into a computing task, and the computing task is stored into a second work queue, when the data to be processed is incomplete, the incomplete data is stored into a first work queue, and after the data is received completely, the complete data to be processed is encapsulated into a computing task. Wherein, whether the data to be processed is complete is judged according to the pre-configured data protocol. If there is no idle process in the process pool of the computing operation, the computing task can be stored into the second work queue corresponding to the process pool of the computing operation, and consumption of the lock can be avoided.

[0087] In one embodiment, step S20 further comprises:

[0088] When it is judged that the data to be processed is incomplete, it is judged whether there is an idle process in the first process pool, when it is judged that there is no idle process in the first process pool, the data to be processed is stored into the first work queue, and when it is judged that there is an idle process in the first process pool, the incomplete data corresponding to the data to be processed is received by using the idle process.

[0089] When the data to be processed is incomplete, and when there is no idle process in the process pool of the input / output operation, the data to be processed is stored into the first work queue, and the idle process in the process pool of the input / output operation is actively acquired without a lock, so that lock contention for process resources in the process pool of the input / output processing is reduced to a lock-free state, and the problem of low resource utilization due to lock contention is avoided.

[0090] In one embodiment, the method further comprises:

[0091] It is judged whether the first work queue stores data to be processed or computing results to be fed back, when it is judged that neither the first work queue stores data to be processed nor computing results to be fed back, the idle process in the first process pool is set to a preset state. Wherein, the preset state includes a sleep state.

[0092] Step S30: It is judged whether there is an idle process in the second process pool, when it is judged that there is an idle process in the second process pool, the computing task is read from the second work queue by using the idle process, computing processing is performed on the computing task based on a pre-set processing rule to obtain a computing result, and the computing result is stored into the first work queue.

[0093] In the embodiment, it is judged whether there is an idle process in the process pool of the computing operation. When there is an idle process in the process pool of the computing operation, the idle process is used to read the computing task from the second work queue, the computing task is executed based on a preset processing rule to obtain a computing result, and the obtained computing result is stored in the first work queue.

[0094] Due to the computing-intensive characteristics of the computing operation, the process pool of the divided computing operation can continuously process the computing pipeline without interruption in the case of complete to-be-computed data, avoiding the case that the idle process cannot provide services due to the suspension or interruption of the computing pipeline, and avoiding the problem of system resource consumption caused by frequent system resource scheduling due to the suspension or interruption of the process, thereby improving the utilization rate of system resources, including CPU resources, memory resources, network resources, and the like.

[0095] The second work queue corresponding to the process of the computing operation is created, and the computing task ready for computation is stored in the second work queue. The process of the computing operation actively and lock-free acquires the computing task, so that the lock contention for process resources in the process pool of the input and output operation is reduced to a lock-free state, and the non-functional consumption of system resources caused by lock contention is reduced. It should be noted that in the prior art, multiple work processes need to perform certain locking processing when acquiring tasks to prevent repeated processing, and prevent inconsistency or repetition of resources.

[0096] Further, the computing task is executed based on the preset processing rule to obtain a computing result, including:

[0097] The identifier of the computing task is acquired, the computing processing rule corresponding to the computing task is found from a preconfigured mapping relationship table based on the identifier, and the computing task is executed based on the computing processing rule to obtain a computing result.

[0098] In one embodiment, after step S30, the method further includes:

[0099] It is judged whether the second work queue stores a to-be-processed computing task. When it is judged that the second work queue does not store a to-be-processed computing task, the idle process in the second process pool is set to a preset state.

[0100] When it is judged that the second work queue stores a to-be-processed computing task, the to-be-processed computing task is read from the second work queue based on the idle process in the second process pool, and the computing task is executed. The preset state includes a sleep state.

[0101] Step S40: judging whether there is an idle process in the first process pool, when judging that there is an idle process in the first process pool, using the idle process to read the calculation result from the first work queue, and feeding back the calculation result to the client.

[0102] In the embodiment, it is judged whether there is an idle process in the process pool of the input / output operation, if there is, the idle process is used to read the calculation result from the first work queue, and the calculation result is fed back to the client, so that the lock introduced due to concurrency when the process pool acquires a processing task can be eliminated, and resource consumption caused by lock contention can be reduced.

[0103] Due to the waiting characteristics of the input / output operation, the divided input / output operation process pool can support asynchronous callback processing operation based on the asynchronous IO mechanism supported by the underlying operating system, solve the consumption of processes or threads caused by input / output waiting, reduce the consumption of system resources caused by frequent creation, destruction or scheduling of processes or threads under large concurrency, and improve the user's usage rate of system resources. It should be noted that the non-asynchronous IO will wait indefinitely or make the process sleep when the data transfer is not completed, and will continue the subsequent operation only when the data reception is completed, which makes the process resources used by the current IO occupied. The asynchronous IO will use process or thread resources only after the data is received.

[0104] On the divided input / output operation process pool, the corresponding first work queue is set, the input / output ready calculation result is stored in the corresponding first work queue, and the input / output processing process is actively taken without locking, so that the lock contention of process or thread resources in the input / output processing process pool is reduced to a lock-free state, and the low resource utilization caused by lock contention is greatly reduced.

[0105] In one embodiment, step S40 further includes:

[0106] When it is judged that there is no idle process in the first process pool, the calculation result is packaged into a transmission task, the transmission task is stored in the first work queue, and the transmission operation is performed on the transmission task when there is an idle process in the first process pool.

[0107] In addition, the embodiment of the present application further provides a computer readable storage medium, which can be any one or any combination of a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory and the like. The computer readable storage medium comprises a data processing program 10, and the data processing program 10 is executed by a processor to realize the following operations.

[0108] A creating step of dividing a preset process pool into a first process pool for input / output operations and a second process pool for computing operations, and respectively creating a first work queue corresponding to the first process pool and a second work queue corresponding to the second process pool;

[0109] A receiving step of receiving a request for data processing sent by a client, obtaining to-be-processed data in the request, judging whether the to-be-processed data is complete based on a preconfigured data protocol, when the to-be-processed data is complete, encapsulating the to-be-processed data into a computing task, and storing the computing task to the second work queue;

[0110] A processing step of judging whether there is an idle process in the second process pool, when it is judged that there is an idle process in the second process pool, reading the computing task from the second work queue by using the idle process, performing computing processing on the computing task based on a preset processing rule to obtain a computing result, and storing the computing result to the first work queue; and

[0111] A feedback step of judging whether there is an idle process in the first process pool, when it is judged that there is an idle process in the first process pool, reading the computing result from the first work queue by using the idle process, and feeding back the computing result to the client.

[0112] The specific implementation of the computer readable storage medium of the present application is basically the same as that of the above-mentioned data processing method, and will not be repeated here.

[0113] It should be noted that the above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments. Moreover, the terms "include", "contain" or any other variant thereof in this paper are intended to cover non-exclusive inclusion, so that the process, device, article or method including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, device, article or method. Without more limitation, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, device, article or method including the element.

[0114] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and the necessary general hardware platform, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk) as described above, and includes a plurality of instructions for causing an end device (which can be a mobile phone, a computer, an electronic device, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0115] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data processing method applied to an electronic device, characterized in that, The method comprises: a creating step of dividing a preset process pool into a first process pool for input / output operations and a second process pool for computing operations, and creating a first work queue corresponding to the first process pool and a second work queue corresponding to the second process pool respectively; a receiving step of receiving a request for data processing sent by a client, obtaining to-be-processed data in the request, judging whether the to-be-processed data is complete based on a pre-configured data protocol, encapsulating the to-be-processed data as a computing task when the to-be-processed data is complete, and storing the computing task to the second work queue; a processing step of judging whether there is an idle process in the second process pool, reading the computing task from the second work queue by using the idle process when it is judged that there is an idle process in the second process pool, determining a computing processing rule corresponding to the computing task based on an identifier of the computing task, performing computing processing on the computing task based on the computing processing rule to obtain a computing result, and storing the computing result to the first work queue; and a feedback step of judging whether there is an idle process in the first process pool, reading the computing result from the first work queue by using the idle process when it is judged that there is an idle process in the first process pool, and feeding back the computing result to the client, and encapsulating the computing result as a transmission task when it is judged that there is no idle process in the first process pool, storing the transmission task to the first work queue, and performing a transmission operation on the transmission task when there is an idle process in the first process pool.

2. The data processing method of claim 1, wherein, The receiving step further comprises: when it is judged that the to-be-processed data is incomplete, judging whether there is an idle process in the first process pool, storing the to-be-processed data to the first work queue when it is judged that there is no idle process in the first process pool, and receiving incomplete data corresponding to the to-be-processed data by using the idle process when it is judged that there is an idle process in the first process pool.

3. The data processing method of claim 2, wherein, After the feedback step, the method further comprises: judging whether the first work queue stores to-be-processed data or to-be-fed-back computing result, and setting an idle process in the first process pool to a preset state when it is judged that the first work queue does not store to-be-processed data and to-be-fed-back computing result.

4. The data processing method of claim 1, wherein, After the processing step, the method further comprises: judging whether the second work queue stores to-be-processed computing task, and setting an idle process in the second process pool to a preset state when it is judged that the second work queue does not store to-be-processed computing task; when it is judged that the second work queue stores to-be-processed computing task, reading the to-be-processed computing task from the second work queue based on the idle process in the second process pool, and performing computing processing on the computing task.

5. The data processing method according to any one of claims 1 to 4, characterized in that, The determining of the computing processing rule corresponding to the computing task based on the identifier of the computing task comprises: obtaining the identifier of the computing task, and searching for the computing processing rule corresponding to the computing task from a pre-configured mapping relationship table based on the identifier.

6. A data processing apparatus, characterized by The device comprises: The creating module divides a preset process pool into a first process pool for input / output operations and a second process pool for computing operations, and respectively creates a first work queue corresponding to the first process pool and a second work queue corresponding to the second process pool; The receiving module receives a request for data processing sent by a client, obtains to-be-processed data in the request, judges whether the to-be-processed data is complete based on a preconfigured data protocol, encapsulates the to-be-processed data into a computing task when the to-be-processed data is complete, and stores the computing task to the second work queue; The processing module judges whether there is an idle process in the second process pool, reads the computing task from the second work queue by using the idle process when it is judged that there is an idle process in the second process pool, determines a computing processing rule corresponding to the computing task based on an identifier of the computing task, performs computing processing on the computing task based on the computing processing rule to obtain a computing result, and stores the computing result to the first work queue; and The feedback module judges whether there is an idle process in the first process pool, reads the computing result from the first work queue by using the idle process when it is judged that there is an idle process in the first process pool, and feeds back the computing result to the client, encapsulates a transmission task when it is judged that there is no idle process in the first process pool, stores the transmission task to the first work queue, and performs a transmission operation on the transmission task when there is an idle process in the first process pool.

7. The data processing apparatus of claim 6, wherein, The receiving module further includes: When it is judged that the to-be-processed data is incomplete, it is judged whether there is an idle process in the first process pool, the to-be-processed data is stored to the first work queue when it is judged that there is no idle process in the first process pool, and the incomplete data corresponding to the to-be-processed data is received by using the idle process when it is judged that there is an idle process in the first process pool.

8. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected with the at least one processor in communication; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data processing method of any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a data processing program, and the data processing program is executed by the processor to implement the steps of the data processing method of any one of claims 1 to 4.

Images