A data processing method and device

Abstract

The present invention discloses a data processing method and device, and relates to the field of computer technology. A specific implementation of the method includes: the service of the main thread receives request data, and performs first data processing corresponding to the request data; when the request data is the first request data, the service of the main thread persists the result data obtained by the first data processing to the local storage device through the memory file queue component, and the first request data is the request data that needs to perform a first type of resource operation during the processing; after the background listener listens to the persisted result data, it performs the first type of resource operation according to the result data to perform a second data processing corresponding to the first request data, and the background listener is independent of the main thread service. This implementation method can fundamentally improve the overall Ops of the cluster, improve the optimization effect of the imbalance of resource access speed, does not require too much hardware resource investment, and the data is more secure.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] In the process of service interface development, various external resources are often involved, such as redis (an open source distributed cache framework with multiple cache containers for data access operations), MQ (message queue, a pipeline for message queuing processing), etc. When accessing these external resources, some resources are fast, such as redis; some resources are slow, such as MQ production. When these resources are integrated into the service, slow resource access will lead to low overall Ops (operation per second, the number of request operations processed per second, which is one of the basic indicators for measuring cluster performance), that is, the short board effect. Some resources in the service are accessed quickly, while some are accessed slowly, resulting in an imbalance in speed matching.

[0003] There are currently two solutions to the imbalance in resource access speed: one is asynchronous tuning, that is, using the thread pool to asynchronously process slow resources, but this optimization is useless when there is high concurrency and the data is very important, because after the thread pool is full, the main thread will still be used to perform slow resource access operations, which cannot fundamentally improve the overall Ops; the second is slow resource performance optimization, that is, tuning slow resources so that they no longer become bottlenecks. Taking MQ production as an example, MQ synchronous production performance is poor. When high concurrent traffic comes, production performance will become worse and worse. MQ team members are needed to help add brokers with less pressure and better performance (used in MQ to provide functions such as receiving, storing, and pulling messages) to solve the problem. As the traffic increases, more and more brokers are added, and the requirements for hardware resources are very high, and sufficient resources need to be allocated. However, in many cases, these broker resources are shared and shared by multiple businesses, so the optimization is not satisfactory.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] It is unable to fundamentally improve the overall Ops of the cluster, has poor optimization effect on the imbalance of resource access speed, and requires too much hardware resource investment. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a data processing method and device, which can fundamentally improve the overall Ops of the cluster, improve the optimization effect of the imbalance of resource access speed, do not require too much hardware resource investment, and make the data more secure.

[0007] To achieve the above objective, according to one aspect of an embodiment of the present invention, a data processing method is provided.

[0008] A data processing method includes: a main thread service receives request data and performs first data processing corresponding to the request data; when the request data is first request data, the main thread service persists result data obtained by the first data processing to a local storage device through a memory file queue component, and the first request data is request data that requires a first type of resource operation during the processing; after a background listener listens to the persisted result data, the background listener executes the first type of resource operation according to the result data to perform second data processing corresponding to the first request data, wherein the background listener is independent of the main thread service.

[0009] Optionally, the result data obtained by processing the first data is persisted to a local storage device through a memory file queue component, including: storing the result data in a memory queue through the memory file queue component, and when the memory queue meets the trigger condition for the data disk operation, persisting the result data from the memory queue to the file queue of the local storage device.

[0010] Optionally, when the memory queue meets the trigger condition for the data disk write operation, the result data is persisted from the memory queue to the file queue of the local storage device, including: when at least one of the following conditions is met: the number of data items in the memory queue is greater than a quantity threshold and the current distance from the last execution of the persistence operation is greater than a duration threshold, triggering the operation of persisting the result data from the memory queue to the file queue of the local storage device.

[0011] Optionally, the service of the main thread persists the result data from the memory queue to the file queue of the local storage device by means of zero copy and memory file mapping through the memory file queue component.

[0012] Optionally, it also includes: when the request data is second request data, the service of the main thread executes a second type of resource operation according to the result data to perform a third data processing corresponding to the second request data, and the second request data is request data that requires the second type of resource operation to be performed during the processing.

[0013] Optionally, it also includes: the service of the main thread determines whether the request data is the first request data or the second request data according to a resource operation speed corresponding to a resource that needs to be operated during the processing of the request data.

[0014] Optionally, the background listener performs the first type of resource operation through a sub-thread independent of the main thread, and the number of the sub-threads can be dynamically increased or decreased.

[0015] According to another aspect of an embodiment of the present invention, a data processing device is provided.

[0016] A data processing device includes: a service module, which is used to receive request data and perform first data processing corresponding to the request data; when the request data is first request data, persisting result data obtained by the first data processing to a local storage device through a memory file queue component, and the first request data is request data that requires a first type of resource operation during the processing; a background listener, which is used to execute the first type of resource operation according to the result data after listening to the persisted result data, so as to perform second data processing corresponding to the first request data, wherein the background listener is independent of the service module.

[0017] Optionally, the service module includes a persistence module, which is used to: store the result data in a memory queue through the memory file queue component, and when the memory queue meets the trigger condition for the data disk operation, persist the result data from the memory queue to the file queue of the local storage device.

[0018] Optionally, the persistence module is also used to trigger an operation of persisting the result data from the memory queue to the file queue of the local storage device when at least one of the following conditions is met: the number of data items in the memory queue is greater than a quantity threshold and the duration from the current time to the last execution of the persistence operation exceeds a duration threshold.

[0019] Optionally, the persistence module is further used to: persist the result data from the memory queue to the file queue of the local storage device by means of zero copy and memory file mapping through the memory file queue component.

[0020] Optionally, the service module is also used to: when the request data is second request data, execute a second type of resource operation according to the result data to perform a third data processing corresponding to the second request data, and the second request data is request data that requires the second type of resource operation to be performed during the processing.

[0021] Optionally, the service module determines whether the request data is the first request data or the second request data according to a resource operation speed corresponding to a resource that needs to be operated during the processing of the request data.

[0022] Optionally, the background listener executes the first type of resource operation through a sub-thread independent of the main thread, and the number of the sub-threads can be dynamically increased or decreased, and the main thread is the thread of the service module.

[0023] According to yet another aspect of the embodiments of the present invention, an electronic device is provided.

[0024] An electronic device comprises: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method provided in an embodiment of the present invention.

[0025] According to yet another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0026] A computer readable medium stores a computer program, which, when executed by a processor, implements a data processing method provided by an embodiment of the present invention.

[0027] An embodiment of the above invention has the following advantages or beneficial effects: the service of the main thread receives the request data and performs the first data processing corresponding to the request data. When the request data is the first request data, the result data obtained by the first data processing is persisted to the local storage device through the memory file queue component; after the background listener listens to the persisted result data, the first type of resource operation is performed according to the result data to perform the second data processing corresponding to the first request data. There is no need to bother to perform asynchronous service tuning. Since the background listener exists independently of the service of the main thread, it will not affect the performance of the overall cluster, and can fundamentally improve the overall Ops of the cluster and improve the optimization effect of the imbalance of resource access speed. Since the memory file queue component itself depends on the scheduling behavior of the system, the demand for hardware resources is greatly reduced, so there is no need to invest too much hardware resources. And since the result data of the first data processing is persisted to the local storage device, the restarted machine will continue the data processing, so that the user data will not be discarded due to the restart of the machine, and the data is safer.

[0028] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0030] Figure 1 is a schematic diagram of main steps of a data processing method according to an embodiment of the present invention;

[0031] Figure 2is a schematic diagram of the execution flow of the main thread service according to an embodiment of the present invention;

[0032] Figure 3 is a processing logic diagram of a memory file queue component according to an embodiment of the present invention;

[0033] Figure 4 is a schematic diagram of the execution flow of a background listener according to an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0035] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0036] Figure 7 It is a schematic diagram of the structure of a computer system suitable for implementing a server of an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] Figure 1 It is a schematic diagram of main steps of a data processing method according to an embodiment of the present invention.

[0039] like Figure 1 As shown, a data processing method according to an embodiment of the present invention mainly includes the following steps S101 to S103.

[0040] Step S101: The service of the main thread receives request data and performs first data processing corresponding to the request data.

[0041] Step S102: When the request data is first request data, the service of the main thread persists the result data obtained by processing the first data to the local storage device through the memory file queue component. The first request data is request data that requires a first type of resource operation during the processing.

[0042] Step S103: after listening to the persisted result data, the background listener executes a first type of resource operation according to the result data to perform second data processing corresponding to the first request data, wherein the background listener is independent of the service of the main thread.

[0043] In one embodiment, when the request data of the main thread is the second request data, the service executes the second type of resource operation according to the above result data to perform the third data processing corresponding to the second request data, and the second request data is the request data that needs to perform the second type of resource operation during the processing.

[0044] The first type of resource operation may be a slow resource operation, such as MQ production.

[0045] The second type of resource operation may be a fast resource operation, such as redis access, etc. The service of the main thread may determine whether the request data is the first request data or the second request data according to the resource operation speed corresponding to the resource that needs to be operated during the processing of the request data.

[0046] In the process of processing request data, the services of the cluster often involve various external resources, such as redis, MQ, etc. When accessing these external resources, some resources are fast, such as redis, and some are slow, such as MQ production. According to the resource operation speed corresponding to the resources that need to be operated during the processing of the request data, the service of the main thread can determine whether the request data is the first request data or the second request data.

[0047] For example, if the request data needs to operate slow resources during data processing, the request data is the first request data, and if the request data needs to operate fast resources during data processing, the request data is the second request data. The resources that the request data needs to operate are determined according to business needs, and the speed of resource operation corresponding to a certain resource is related to the resource itself. According to business needs, different request data correspond to their own processing logic. The processing logic of some request data includes the logic of fast resource operation. In this case, the processing process requires the operation of fast resources (the corresponding resource operation speed is fast), and the request data is the first request data. Those skilled in the art can refer to the above method to determine whether the request data is the second request data.

[0048] The service of the main thread is the service responsible for data processing in the cluster. The embodiment of the present invention decouples the operation of processing slow resources from the service of the main thread, and executes it by a background listener independent of the service of the main thread, thereby avoiding the slow resource operation leading to low overall cluster Ops (operation per second, the number of request operations processed per second, which is one of the basic indicators for measuring cluster performance), thereby overcoming the short board effect.

[0049] The background listener is a post-worker, which is a new application different from the main thread service. It is used to listen to the persistent memory file queue data, that is, the persistent result data of the first data processing. For example, when the result data is listened to, the MQ production and sending MQ operations are performed according to the result data, and the data in the MQ is the result data.

[0050] The first data processing is the process of performing specific business processing according to the request data after the service receives the request data. Specifically, it can be the initial one or more processing links in the request data processing process. The initial one or more processing links do not involve slow resource operations.

[0051] The background listener performs the first type of resource operations through a sub-thread independent of the main thread, and the number of sub-threads can be dynamically increased or decreased to control the acceleration or slowing down of slow resource operations. Since the thread of the background listener is independent of the main thread, the acceleration and slowing down of its slow resource operations will not affect the execution of the main thread, so that the overall cluster no longer has the problem of speed imbalance, so the overall Ops of the cluster is also improved.

[0052] In one embodiment, the result data obtained by the first data processing is persisted to the local storage device through the memory file queue component, including: storing the result data in the memory queue through the memory file queue component, and when the memory queue reaches the trigger condition for the data disk operation, persisting the result data from the memory queue to the file queue of the local storage device.

[0053] The memory file queue component of the embodiment of the present invention is a component used to persist the data in the memory queue to the local disk. The memory queue is a queue existing in the memory.

[0054] Local storage devices such as local disks.

[0055] The service of the main thread uses the memory file queue component to persist the result data from the memory queue to the file queue of the local storage device by means of zero copy and memory file mapping, which has the advantages of good performance and high processing efficiency.

[0056] In one embodiment, when at least one of the following conditions is satisfied: the number of data in the memory queue is greater than the quantity threshold, or the time from the last persistence operation to the last persistence operation is greater than the time threshold, the operation of persisting the result data from the memory queue to the file queue of the local storage device is triggered. For example, whenever the number of data in the memory queue is greater than 10,000 and data needs to be written to the disk (i.e., persisted to the local disk), each time the data is written to the disk for more than 10ms (milliseconds), the data will also be written to the disk.

[0057] Figure 2It is a schematic diagram of the execution flow of the main thread service according to an embodiment of the present invention.

[0058] like Figure 2 As shown, in the execution process of the main thread service (referred to as service) of an embodiment of the present invention, the slow resource operation (first type of resource operation) takes sending MQ as an example. The process includes: after the service receives the request data, it performs the first data processing corresponding to the request data. The first data processing is the process of performing specific business processing according to the request data after the service receives the request data. Taking the request data as an inventory deduction request as an example, the first data processing can be the processing of deducting the existing inventory according to the inventory deduction request, and the result data obtained is the inventory data after the deduction.

[0059] After performing the first data processing, the service determines whether it needs to send MQ. If so, it means that the request data is the first request data that requires slow resource operations during the processing process. In this case, data production is performed and the data is stored in the memory queue. Otherwise, it means that the request data is the second request data that requires fast resource operations during the processing process. In this case, the service continues with subsequent data processing corresponding to the second request data, that is, the third data processing.

[0060] Figure 2 The first data processing and the third data processing are uniformly represented by normal data processing, which refers to data processing performed by the service and does not involve slow resource operations. Those skilled in the art can understand that Figure 2 In the flowchart shown, the normal data processing executed at the beginning of the process is the first data processing; after determining whether MQ needs to be sent and the determination result is "no", the normal data processing executed is the third data processing.

[0061] In the above-mentioned operations of data production and storing data into the memory queue, data production means obtaining the result data of the first data processing, such as obtaining the inventory data after deduction. Storing data into the memory queue means storing the result data into the memory queue, such as storing the inventory data after deduction into the memory queue.

[0062] After the result data of the first data processing is stored in the memory queue, the timing starts, and it is determined whether at least one of the following two trigger conditions for the data storage operation is met: the number of data in the memory queue is greater than the quantity threshold; the current time from the last persistence operation (i.e. Figure 2 The current time difference in the time interval (i.e., the current time difference from the last time data was stored on disk (or persisted)) exceeds the time threshold; if at least one trigger condition is met, data storage is triggered. For example, when the number of data items exceeds 10,000, data storage is triggered; when the current time difference (i.e., the time difference from the last time data was stored on disk (or persisted)) exceeds 10ms (milliseconds), data storage is also triggered.

[0063] The embodiment of the present invention improves the existing MQ sending logic. In the existing MQ sending logic, after the service performs the first data processing corresponding to the request data, the service directly starts the MQ production sending behavior. However, in the embodiment of the present invention, after the service performs the first data processing corresponding to the request data, the result data of the first data processing is stored in the memory file queue. Specifically, the result data is first stored in the memory queue and then persisted in the file queue of the local disk. The background listener listens to the file queue and consumes the persisted result data to perform the operations of generating and sending MQ. Therefore, the embodiment of the present invention realizes the decoupling of MQ slow resource operations from the service, overcomes the "short board effect" brought by slow resource operations to cluster performance, and solves the speed imbalance caused by operations such as fast and slow resource access within the service.

[0064] The embodiment of the present invention uses the memory file queue component to persist the result data obtained by the first data processing to the local disk. The processing logic of the memory file queue component is as follows: Figure 3 As shown, after the result data obtained by the first data processing is stored in the memory queue, if at least one of the following conditions is satisfied: the number of data in the memory queue is greater than 10,000, and the current time difference (i.e., the time difference from the current time to the last time data was dropped to the disk) exceeds 10 milliseconds, the data drop operation of the memory queue is triggered, that is, the data files in the memory queue are persisted to the disk for storage, so that the post-worker (i.e., the background listener) can consume and process these data. Since the memory file queue component uses zero copy and memory file mapping to persist the result data from the memory queue to the file queue of the local storage device, and all of them are local data operations, the performance of pressing data (storing data) to the memory file queue (a general term for memory queues and file queues) is much better than the performance of pressing data to MQ. In addition, the memory file queue component relies on the scheduling behavior of the operating system itself, which can avoid disk copying, so that the speed of processing disk mapped data is the same as processing in memory, which greatly improves data processing performance.

[0065] Figure 4 It is a schematic diagram of the execution flow of a background listener according to an embodiment of the present invention.

[0066] like Figure 4As shown, the background listener of one embodiment of the present invention is a post-worker, and its execution process is as follows: the post-worker monitors whether there is persistent result data in the file queue of the local disk, and the result data is obtained by performing the first data processing corresponding to the request data, such as the deducted inventory data persisted in the local disk file queue. If the result data is in the local disk file queue, the result data is put into MQ and the MQ is sent to perform subsequent data processing (i.e., the second data processing). If there is no result data in the local disk, the file queue continues to be monitored.

[0067] This post-worker is usually placed in a separate sub-thread for processing, and exists completely independently of the main thread. The purpose of speeding up or slowing down MQ production is achieved by increasing or reducing multiple consumer threads (i.e., threads of the post-worker). Due to the independence of this consumer worker (i.e., post-worker), the speed of its production will not affect the execution of the main thread. Since the embodiment of the present invention separates the slow resource operation from the service of the main thread, the slow resource access is separated from the main thread, so that the overall cluster no longer has the problem of speed imbalance, and the overall Ops is also significantly improved.

[0068] In the embodiment of the present invention, when the service receives the request data and after the first data processing, when the slow resource needs to be operated, the result data of the first data processing is first pressed (stored) into the memory file queue to be persisted on the disk, and then the post-worker is used to process the access to the slow resource. In this way, the slow resource operation is separated from the service by the asynchronous worker, so that the slow resource operation will not affect the overall cluster performance. Therefore, compared with the prior art, the embodiment of the present invention has the following advantages: no need to perform asynchronous service tuning, because the post-worker exists independently of the service itself, and will not affect the performance of the overall cluster; no need to invest too much hardware resources, because the memory file queue itself depends on the scheduling behavior of the system, the demand for hardware resources is greatly reduced; data is safer, when some machines in the service crash, the data required for subsequent data processing is persisted to the local disk by the memory file queue component, so that the restarted machine will continue the data processing, so that the user data will not be discarded due to the restart of the machine.

[0069] Figure 5 4 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention.

[0070] like Figure 5 As shown, a data processing device 500 according to an embodiment of the present invention mainly includes: a service module 501 and a background listener 502 .

[0071] Service module 501 is used to receive request data and perform first data processing corresponding to the request data; when the request data is first request data, the result data obtained by the first data processing is persisted to the local storage device through the memory file queue component. The first request data is request data that requires a first type of resource operation during the processing.

[0072] The background listener 502 is used to perform a first type of resource operation according to the result data after listening to the persistent result data, so as to perform a second data processing corresponding to the first request data, wherein the background listener is independent of the service module.

[0073] In one embodiment, the service module includes a persistence module, which is used to: store the result data in the memory queue through the memory file queue component, and when the memory queue meets the trigger condition for the data disk operation, persist the result data from the memory queue to the file queue of the local storage device.

[0074] In one embodiment, the persistence module is used to trigger the operation of persisting the result data from the memory queue to the file queue of the local storage device when at least one of the following conditions is met: the number of data items in the memory queue is greater than a quantity threshold, or the time from the current time to the last time a persistence operation is performed exceeds a time threshold.

[0075] The persistence module can be specifically used to persist the result data from the memory queue to the file queue of the local storage device by using the memory file queue component and the zero copy and memory file mapping method.

[0076] The service module 501 can also be used to: when the request data is second request data, execute a second type of resource operation according to the result data to perform a third data processing corresponding to the second request data, and the second request data is request data that requires a second type of resource operation during the processing.

[0077] In one embodiment, the service module 501 determines whether the request data is the first request data or the second request data according to the resource operation speed corresponding to the resource that needs to be operated during the processing of the request data.

[0078] The background listener 502 can perform the first type of resource operation through a sub-thread independent of the main thread, and the number of sub-threads can be dynamically increased or decreased. The main thread is the thread of the service module.

[0079] In addition, the specific implementation content of the data processing device described in the embodiment of the present invention has been described in detail in the data processing method described above, so the repeated content will not be described again here.

[0080] Figure 6An exemplary system architecture 600 is shown to which the data processing method or data processing device according to the embodiment of the present invention can be applied.

[0081] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0082] Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0083] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0084] The server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as product information - only an example) to the terminal device.

[0085] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the data processing device is generally disposed in the server 605 .

[0086] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0087] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 suitable for implementing a server of an embodiment of the present application. Figure 7 The server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0088] like Figure 7As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0089] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0090] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present application are executed.

[0091] It should be noted that the computer-readable medium shown in the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0092] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0093] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor, for example, may be described as: a processor including a service module and a background listener. The names of these modules do not, in certain cases, constitute limitations on the modules themselves. For example, the service module may also be described as "a module for receiving request data and performing first data processing corresponding to the request data; when the request data is the first request data, persisting the result data obtained by the first data processing to a local storage device through a memory file queue component".

[0094] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: the service of the main thread receives request data and performs a first data processing corresponding to the request data; the service of the main thread, when the request data is the first request data, persists the result data obtained by the first data processing to the local storage device through the memory file queue component, and the first request data is the request data that needs to perform a first type of resource operation during the processing; after the background listener listens to the persisted result data, it performs the first type of resource operation according to the result data to perform a second data processing corresponding to the first request data, wherein the background listener is independent of the service of the main thread.

[0095] According to the technical solution of the embodiment of the present invention, the service of the main thread receives the request data and performs the first data processing corresponding to the request data. When the request data is the first request data, the result data obtained by the first data processing is persisted to the local storage device through the memory file queue component; after the background listener listens to the persisted result data, it performs the first type of resource operation according to the result data to perform the second data processing corresponding to the first request data. There is no need to bother with asynchronous service tuning. Since the background listener exists independently of the service of the main thread, it will not affect the performance of the overall cluster, and can fundamentally improve the overall Ops of the cluster and improve the optimization effect of the imbalance of resource access speed. Since the memory file queue component itself depends on the scheduling behavior of the system, the demand for hardware resources is greatly reduced, so there is no need to invest too much hardware resources. And since the result data of the first data processing is persisted to the local storage device, restarting the machine will continue the data processing, so that user data will not be discarded due to machine restart, and the data is safer.

[0096] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: The service of the main thread receives the request data and performs a first data processing corresponding to the request data; The service of the main thread persists the result data obtained by processing the first data to a local storage device through a memory file queue component when the request data is first request data, wherein the first request data is request data that requires a first type of resource operation during processing; the first type of resource operation is a slow resource operation; After listening to the persisted result data, the background listener executes the first type of resource operation according to the result data to perform second data processing corresponding to the first request data, wherein the background listener is independent of the service of the main thread.

2. The method according to claim 1, characterized in that The step of persisting the result data obtained by the first data processing to a local storage device through a memory file queue component includes: The result data is stored in a memory queue through the memory file queue component, and when the memory queue meets the triggering condition of the data storage operation, the result data is persisted from the memory queue to the file queue of the local storage device.

3. The method according to claim 2, characterized in that When the memory queue meets the triggering condition of the data storage operation, persisting the result data from the memory queue to the file queue of the local storage device includes: When at least one of the following conditions is met: the number of data items in the memory queue is greater than a quantity threshold and the duration from the current time to the last time the persistence operation is performed exceeds a duration threshold, an operation of persisting the result data from the memory queue to the file queue of the local storage device is triggered.

4. The method according to claim 2 or 3, characterized in that: The service of the main thread uses the memory file queue component to persist the result data from the memory queue to the file queue of the local storage device by means of zero copy and memory file mapping.

5. The method according to claim 1, characterized in that Also includes: When the request data is second request data, the service of the main thread executes a second type of resource operation according to the result data to perform a third data processing corresponding to the second request data, and the second request data is request data that requires the second type of resource operation to be performed during the processing.

6. The method according to claim 5, characterized in that Also includes: The service of the main thread determines whether the request data is the first request data or the second request data according to a resource operation speed corresponding to a resource that needs to be operated during the processing of the request data.

7. The method according to claim 1, characterized in that The background listener executes the first type of resource operation through a sub-thread independent of the main thread, and the number of the sub-threads can be dynamically increased or decreased.

8. A data processing device, characterized in that: include: A service module, configured to receive request data and perform first data processing corresponding to the request data; In the case where the request data is first request data, persisting result data obtained by processing the first data to a local storage device through a memory file queue component, wherein the first request data is request data requiring a first type of resource operation during processing; The first type of resource operations are slow resource operations; A background listener is used to execute the first type of resource operation according to the result data after listening to the persisted result data, so as to perform second data processing corresponding to the first request data, wherein the background listener is independent of the service module.

9. The device according to claim 8, characterized in that The service module includes a persistence module, which is used to: The result data is stored in a memory queue through the memory file queue component, and when the memory queue meets the triggering condition of the data storage operation, the result data is persisted from the memory queue to the file queue of the local storage device.

10. The device according to claim 9, characterized in that The persistence module is also used to: When at least one of the following conditions is met: the number of data items in the memory queue is greater than a quantity threshold and the duration from the current time to the last time the persistence operation is performed exceeds a duration threshold, an operation of persisting the result data from the memory queue to the file queue of the local storage device is triggered.

11. The device according to claim 9 or 10, characterized in that The persistence module is also used to: persist the result data from the memory queue to the file queue of the local storage device by means of zero copy and memory file mapping through the memory file queue component.

12. The device according to claim 8, characterized in that: The service module is also used to: when the request data is second request data, execute a second type of resource operation according to the result data to perform a third data processing corresponding to the second request data, and the second request data is request data that requires the second type of resource operation to be performed during the processing.

13. The device according to claim 12, characterized in that The service module determines whether the request data is the first request data or the second request data according to a resource operation speed corresponding to a resource that needs to be operated during the processing of the request data.

14. The device according to claim 8, characterized in that The background listener executes the first type of resource operation through a sub-thread independent of the main thread, and the number of the sub-threads can be dynamically increased or decreased. The main thread is the thread of the service module.

15. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 7.

16. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images